Background: Human breast cancer most frequently originates within a well-defined anatomical structure referred to as the terminal duct lobular unit (TDLU). This structure is endowed with its very own lobular fibroblasts representing one out of two steady-state fibroblast subtypes—the other being interlobular fibroblasts. While cancer-associated fibroblasts (CAFs) are increasingly appreciated as covering a spectrum of perturbed states, we lack a coherent understanding of their relationship—if any—with the steady-state fibroblast subtypes. To address this, we here established two autologous CAF lines representing inflammatory CAFs (iCAFs) and myofibroblast CAFs (myCAFs) and compared them with already established interlobular- and lobular fibroblasts with respect to their origin and impact on tumor formation. Methods: Primary breast tumor-derived CAFs were transduced to express human telomerase reverse transcriptase (hTERT) and sorted into CD105^low and CD105^high populations using fluorescence-activated cell sorting (FACS). The two populations were tested for differentiation similarities to iCAF and myCAF states through transcriptome-wide RNA-Sequencing (RNA-Seq) including comparison to an available iCAF-myCAF cell state atlas. Inference of origin in interlobular and lobular fibroblasts relied on RNA-Seq profiles, immunocytochemistry and growth characteristics. Osteogenic differentiation and bone formation assays in culture and in vivo were employed to gauge for origin in bone marrow-derived mesenchymal stem cells (bMSCs). Functional characteristics were assessed with respect to contractility in culture and interaction with tumor cells in mouse xenografts. The cells’ gene expression signatures were tested for association with clinical outcome of breast cancer patients using survival data from The Cancer Genome Atlas database. Results: We demonstrate that iCAFs have properties in common with interlobular fibroblasts while myCAFs and lobular fibroblasts are related. None of the CAFs qualify as bMSCs as revealed by lack of critical performance in bone formation assays. Functionally, myCAFs and lobular fibroblasts are almost equally tumor promoting as opposed to iCAFs and interlobular fibroblasts. A myCAF gene signature is found to associate with poor breast cancer-specific survival. Conclusions: We propose that iCAFs and myCAFs originate in interlobular and lobular fibroblasts, respectively, and more importantly, that the tumor-promoting properties of lobular fibroblasts render the TDLU an epicenter for breast cancer evolution.