I took Calculus I in college from a black female instructor, and it was pretty effing difficult!
It is not that hard, but does require a mental discipline that the urban culture is incompatible with. I had no trouble with trig or geometry. I only started stumbling years later with vector calculus.
But then, I was working full time and trying to get a degree at night, not the best way to learn advanced math.