What about the freed blacks that had slaves themselves?
Maybe their white parts could pay back their african parts.
Anthony Johnson (1600-1670) comes to mind. In fact, he was the first slaveholder in the colonies.