My ancestors were spaghetti benders in southern Italy until 1906 and when the romans took slaves they were color blind.
Slaveholders in America North and South started as color blind but Negroes held up better and cost less. There were many local Indian slaves in the Southern colonies/states until it was obviously just not economical. They expired quickly. There were white slaves from Ireland. Most were contracted for a temporary slavery as indentured servants (for some it was a long temporary) and some were outright slaves. Africans were cheaper because the Arabs and local chiefs and kings in Africa were prolific producers. The trade was well developed and there were African slaves in Europe centuries before there was an America and the Arab lands had copious quantities of them and still trade them and employ them and had to return to the well a lot because they castrated the men.