I didn't either... but then until recently we never bothered to care if someone was black/white/brown/whatever.
The first mention of Hank Aaron that I recall was in an Elvin Bishop song. For whatever reason, I assumed he was white.
But your point is excellent. Back then, who cared what color he was? People cared about his skill, and that was about it.