If you're going back past the first comers to America, you're right, the chances of some slaveholding somewhere in your family history is pretty much 100%. The owning of slaves has been prevalent throughout history.
But as I've stated to others here, you don't even have to go back past America's founding. Most everyone, white or black, has some connection to slaveholders right here in America.
In my family, for example, I have records of my forebear Joel Estes, the discoverer and first settler of Estes Park, Colorado, selling his last slaves off in Missouri in 1863. And the family lines going back through Virginia which are full of slaveholders are many.
We’re probably distant cousins. I have Estes in my family tree also.