Tissue-resident macrophages are the most abundant immune cell population in healthy adipose tissue. Adipose tissue macrophages (ATMs) change during metabolic stress and are thought to contribute to metabolic syndrome. Here, we studied ATM subpopulations in steady state and in response to nutritional and infectious challenges. We found that tissue-resident macrophages from healthy epididymal white adipose tissue (eWAT) tightly associate with blood vessels, displaying very high endocytic capacity. We refer to these cells as vasculature-associated ATMs (VAMs). Chronic high-fat diet (HFD) results in the accumulation of a monocyte-derived CD11c+CD64+ double-positive (DP) macrophage eWAT population with a predominant anti-inflammatory/detoxifying gene profile, but reduced endocytic function. In contrast, fasting rapidly and reversibly leads to VAM depletion, while acute inflammatory stress induced by pathogens transiently depletes VAMs and simultaneously boosts DP macrophage accumulation. Our results indicate that ATM populations dynamically adapt to metabolic stress and inflammation, suggesting an important role for these cells in maintaining tissue homeostasis.