That was because originally not everyone could vote, but districts were apportioned by population, with slaves counting as a portion of a person. That was a political compromise to keep the southern states from having the highest representation in the House. The other options were to count every slave (population would be heavier in the south) or count none of them (population would be heavier in the north). To balance they agreed on fractional representation for some.
Right. My point was that the census takers have always "needed" to know more than just "how many people".
In 1790, white people were grouped by males of militia age, males younger than militia age, and females of any age. ;-)