I have a large working vocabulary because my parents had a large working vocabulary. I think that if you grow up in a household with educated English-speaking parents you do have an advantage on the SAT/ACT that is not purely a function of intelligence, but also environment.
Hard work and dedication to purpose are the things that govern success in life, not Ivy League diplomas.
I on the other hand, grew up in a family with a 3rd grade education and a mother with a 6th grade education. They were both from the Appalachians and spoke fluent hillbilly and nothing else. I scored in the 99th percentile on my SAT back in the day. My brother became a civil engineer and I somehow became a nuclear chemist.
By the way, though "uneducated" my parents were some of the most intelligent people I have ever known.
Look at the children of smart non-English speaking immigrants who grow to ace the SAT and you will recognize that smart students climb either way. Is it perhaps not absolutely 100.0000% precisely fair/accurate? Of course, nothing in life is. But it is much more fair and accurate than anything else we have.