Scavenger receptor B1 (SR-B1), the main receptor for high-density lipoprotein (HDL), is key in preventing atherosclerosis. It removes cholesterol from HDL, returning the lipid-poor lipoprotein to the circulation. To study the mechanisms controlling SR-B1 dynamics at the plasma membrane and its internalization rate, we developed a single-chain variable fragment (ScFv) antibody to image the receptor in live cells and track the behavior of single SR-B1 molecules. Unlike transferrin receptors, cholera-toxin-binding gangliosides, and bulk membrane markers, SR-B1 was internalized only marginally over hours. Plasmalemmal retention was not attributable to its C-terminal PDZ-binding domain or to attachment to the cortical cytoskeleton. Instead, SR-B1 undergoes multimerization into large metastable clusters that, despite being mobile in the membrane, fail to enter endocytic pathways. SR-B1 multimerization was impaired by mutating its C-terminal leucine zipper and by disrupting actin polymerization, causing rapid receptor internalization. Multimerization and plasmalemmal retention are critical for SR-B1 function.