At some point in their lives, invariably someone -- quite often themselves -- has convinced them they "can't do math." And invariably someone -- again, quite often themselves -- has told them that this is OK; That they aren't "math people," that you "don't need math." Primary arithmetic and basic mathematics teachers are the principal enablers of this nonsense.
And as you point out, our society and culture encourage this. It's actually cool to be innumerate.
Mathematical and logical reasoning occurs at specific place in the brain, which has virtually no overlap we know of with other faculties. Except for some very well known and statistically insignificant cases of brain trauma or organic brain damage, everyone possesses this faculty. Like any other capability it's largely a matter of self-discipline, application, practice, and proper instruction. It's time to stop excusing innumeracy. The best place to start is by reassuring the innumerate -- gently but firmly -- that there is no such thing as a functioning person who "can't do math."