Human breast cancer is believed to arise in luminal progenitors within the normal breast. A subset of these are double positive (DP) for basal and luminal keratins and localizes to a putative stem cell zone within ducts. We here present a new protocol based on a combination of CD146 with CD117 and CD326 which provides an up to thirty fold enrichment of the DP cells. We show by expression profiling, colony formation, and morphogenesis that CD146high/CD117high/CD326high DP cells belong to a luminal progenitor compartment. While these DP cells are located quite uniformly in ducts, with age a variant type of DP (vDP) cells, which is mainly CD146-negative, accumulates in lobules. Intriguingly, in specimens with BRCA1 mutations known to predispose for cancer, higher frequencies of lobular vDP cells are observed. We propose that vDP cells are strong candidates for tracing the cellular origin of breast cancer.