Many who did own slaves only had one or two and they were more like apprentices in small businesses, many of them used their skills to have their own business after being freed. The slave owning master of a large plantation with a village of slaves was an incredibly small number.
and even a few of them were black