Fibroblasts display extensive transcriptional heterogeneity, yet functional annotation and characterization of their heterocellular relationships remains incomplete. Using mass cytometry, we chart the stromal composition of 18 murine tissues and 5 spontaneous tumor models, with an emphasis on mesenchymal phenotypes. This analysis reveals extensive stromal heterogeneity across tissues and tumors, and identifies coordinated relationships between mesenchymal and immune cell subsets in pancreatic ductal adenocarcinoma. Expression of CD105 demarks two stable and functionally distinct pancreatic fibroblast lineages, which are also identified in murine and human healthy tissues and tumors. Whereas CD105-positive pancreatic fibroblasts are permissive for tumor growth in vivo, CD105-negative fibroblasts are highly tumor suppressive. This restrictive effect is entirely dependent on functional adaptive immunity. Collectively, these results reveal two functionally distinct pancreatic fibroblast lineages and highlight the importance of mesenchymal and immune cell interactions in restricting tumor growth.