Most slaves didn't live on a plantation. Thomas Jackson, for example, had as many as 8 or 9 at a time and he was a college instructor. The average slave owner had a handful of slaves and that indicates most were not plantation owners. And why wouldn't Northerners invest in slaves and enjoy the benefits of the return they might get on their chattel...unless they were opposed to the practice to begin with?
It's misleading -- your specialty -- to speak of averages when the distribution of slave ownership was so strongly skewed. Better to speak of medians, or better still, to offer more datapoints. How many slave-holders had more than 100 slaves? How many had more than 200? 1000? And so on.
The reason that these numbers are important is that there was a social fault line separating large planters, who owned many hundreds of slaves, from the freeholders most of whom owned no slaves, or who, if they did own a slave, owned one or two, who lived not in quarters but with the family.
Big difference, which guys like you are at pains to blur and conceal, the better to work your grift of trying to paint all Southerners, dead and living, as slavers <hisssssss!!>, scum of the earth and just repositories of your undying enmity and spite.