There has been great interest in the magnetic behavior of pyrochlore oxides with the general formula A2B2O7, in which rare-earth (A) and transition metal (B) cations are ordered on separate interpenetrating lattices of corner-sharing tetrahedra. Such materials exhibit behaviors including quantum spin-ice, (quantum) spin-liquid, and ordered magnetic ground states. Yb2Ti2O7 lies on the boundary between a number of competing magnetic ground states. Features in the low-temperature specific heat capacity that vary in sharpness and temperature from sample to sample suggest that, in some cases, the magnetic moments order, while in others, the moments remain dynamic down to temperatures as low as ∼16 mK. In this paper, three different Yb2Ti2O7 samples, all grown by the optical floating zone technique but exhibiting quite different heat capacity behavior, are studied by aberration-corrected scanning transmission microscopy (STEM). Atomic-scale energy-dispersive x-ray analysis shows that a crystal with no specific heat anomaly has substitution of Yb atoms on Ti sites (stuffing). We show that the detailed intensity distribution around the visible atomic columns in annular dark field STEM images is sensitive to the presence of nearby atoms of low atomic number (in this case oxygen) and find significant differences between the samples that correlate both with their magnetic behavior and measurements of Ti oxidation state using electron energy loss spectroscopy. These measurements support the view that the magnetic ground state of Yb2Ti2O7 is extremely sensitive to disorder.