You make a good point. Most of our ancestors came here after slavery was abolished.
The blacks have ancestors who were here before slavery was abolished and many of them - including thousands of blacks - were slave owners.
Also, many Catholics and Jews didn't marry Protestants, so there's that too.
Indians owned slaves, even before the Europeans got here, so some people with an Indian ancestor, somewhere in their family tree, are connected someway, to a slave owner.