Black race compared to white race is associated with more advanced stage and biologically aggressive breast cancer. Consequently, black patients are more frequently treated with neoadjuvant chemotherapy (NAC) than white patients. However, it is unclear how distant recurrence-free survival (DRFS) of black patients treated with NAC, compares to DRFS of black patients treated with adjuvant chemotherapy (AC). We evaluated the association between race, distant recurrence, and type of chemotherapy (AC or NAC) in localized or locally advanced breast cancer. We evaluated DRFS in 807 patients, including 473 black, 252 white, 56 Hispanic, and 26 women of other or mixed race. The association between AC or NAC and DRFS was examined using multivariate Cox proportional hazard models that included race, age, stage, estrogen receptor (ER) and triple negative (TN) status. When the black and white subjects were pooled for the analysis the features associated with worse DRFS included stage III disease and age < 50 years, but not ER-negative disease, TN disease, the use of NAC, or black race. However, in the analysis stratified by race NAC was associated with worse DRFS compared to AC in black (HR 2.70; 95% CI 1.73-4.22; p < 0.0001), but not in white women (HR 1.29, 95% CI 0.56-2.95; p = 0.36). Black patients treated with NAC had worse DRFS than black patients treated with AC, or white patients treated with either NAC or AC. These findings need to be validated in a large-scale observational study and the effect of NAC on the breast cancer microenvironment in black women needs to be further evaluated.