The fugitive slave thing was never a major factor economically and from the Deep South cotton states, hardly a factor at all. There, only those slaves living in seaport towns had any realistic hope of escape and in the Upper South, most of those who escaped were from Maryland and Kentucky --- neither of which were upset about it enough to secede from the Union.
Fugitive slavery did serve as a political club to beat Northern politicians with, but contrary to the Underground Railroad mythology that has grown up in recent decades, very few slaves ever made a successful escape to freedom. Perhaps a few hundred a year (if that) of the 4 million slaves. The 'slave patrols' in most states were quite effective in catching the vast majority of run aways long before they ever got near 'Free' territory.
You’re probably right about the numbers.
What incensed southerners was the fact that northerners would try to prevent them from recapturing their “property.” For some reason this really infuriated them.
In a recent book I read, southerners in northern prisons wrote home about their fears that they would be “exchanged” for a black Union soldier. They would all, quite literally, rather stay in prison or die there than be exchanged for a black man.