Well, technically there is no need to. When they were enslaved, they were not citizens, and therefore had no rights guaranteed by the government.
OOOOOOOOOK. So anyone who happens to land on our soil can be exploited for whatever purposes we wish?
They were held in bondage here because of an immoral government policy.