No, it’s not. It’s ignorant and indifferent parenting, which has it’s roots in culture and dysfunctional families and single parents.
Children learn math skills by playing games; cards, board games, dice games, etc.
That requires interested parents that are willing to invest the time playing games with their children, rather than park them in front of electronic babysitters.
It’s not racism, poverty or genetics. It’s time, lots and lots of time.
Your sentence was partially correct. It’s not time that makes the difference, it’s having interested parents. Being invested in your child’s education is free and has no relationship to socioeconomic status. However, innate intelligence is actually a thing and it’s disingenuous to pretend it isn’t.