It was mostly blacks who enslaved them in the first place.
In this country it was whites that bought most of them