I guess it started with the 1787 3/5 compromise. The South wanted to count its slaves in the tally in order to get a bigger share of congresscritters. Northern states didn't think it was right for the south to get extra congresscritters when a lot of the people being counted weren't even free, and saw the institution of slavery as a national embarassment that must not be encouraged. If the south gathered additional power by counting slaves it would have more votes in congress for the creation of even more slave states. So while a lot of blacks get outraged by being the idea of "being considered 3/5 of a person" the reason wasn't to demean them but rather, to limit the spread of slavery and still keep the states from breaking apart.
I know early census were pretty vague, usually listing by name just the head of houshold and tallying spouse, relatives and slaves generically by age group. After listing everyone by age group there might be a note saying "two are slaves" or some such thing but it's anyone's guess who was who.
Later census records became more specific because it was politicized.