Women and Children were also counted, but couldn't vote. The representatives elected on their behalf were also expected to benefit them.
Oh, and neo-confederates also conveniently neglect to mention that the original reason for the census was for purpose of apportioning tax liabilities. At that time the Slavocracy didn't want their chattel property counted at all.
There was a possibility that was going to happen? I don't think so. The one surety of government is that the people with money will be taxed.
But Indians were not counted in the US census, were not used to apportion representatives & electors, and as non-citizen human beings slaves more closely resembled Indians than wives & children who could not be lawfully sold and were very seldom treated as if slaves.
Further, there were no significant regional differences between North, South, East or West in average numbers of wives & children per voting adult, so counting them all as full citizens (which they were) in no way effected the relative balance of political influence.
But slaves were a very different subject and 3/5 counting represented simply the skill of slave-holders to negotiate better terms than their real numbers would presuppose.