The opposite was, in fact, true.
The feigned apprehension of the Northern Abolitionists and their allies, such as Lincoln, that Southern slaveholders would begin to flock northward and westward with their slaves ignored the clear historical fact that slavery had already died out in the Northern States and that the slave population had shifted almost entirely to the Gulf States.
Josiah J. Evans of South Carolina, stated that slave labor was not suited for the agriculture of the Territories: “There is no pretense that any one of the great staples that constitute the great material of our foreign commerce, can be cultivated anywhere within the limits of these Territories outside of the Territory of Kansas.”
There was absolutely no reason at all for Southern owners to move North with their slaves, and they had no inclination to do so. There was also no real inclination for most slaveholders to migrate into the Territories: They demanded a right which they could not actively use — the legal right to carry slaves where few would or could be taken. The one side fought obsessively for what it was bound to get without fighting; the other, with equal rancor, contended for what in the nature of things it could never use.
Consequently, the whole controversy over the expansion of slavery into the territories became a politically contrived issue.
Slavery was dying in the rest of the world. It had little chance of spreading further into new territories of the continent. It had reached the limits imposed on its expansion by geography and climate, as Kansas, New Mexico, and Utah amply showed. The census of 1860 revealed that there were precisely two slaves in Kansas, and only a handful more in all the remaining territories.
Even the Congressional Republicans had recognized that slavery posed no real threat in the territories, when, early in 1861, they provided for the organization of the new territories of Colorado, Nevada, and Dakota without any ban on slavery.
North and South were not divided by their mutual racism. Slavery was not a genuine issue and there was no need to go to war over it. The men of 1860-1 allowed an academic argument about an imaginary slave in an impossible place to end in a bloody civil war.
If the question was merely one of slavery in the territories, then competent political leadership would have been able to cope with it. Instead, the Northern political class, seeing that the South was steadily becoming a minority in the United States, remained frustrated at the South's ability to cling to power. Not merely was the Northern stand against the threat of slavery in the territories a misdirection, but Northern expressions of moral repugnance towards slavery were totalitarian nonsense.
The Republican party was seeking to push the South into minority status, eliminate Constitutional guarantees of freedom and limit their existence to growing crops.
The purpose of spreading slavery to the south and west was not necessarily to dramatically increase the population of slaves, but to match every new free state with another new slave state.
In Delaware there were less than 2,000 slaves.
But Delaware was a reliable slave state vote in the Senate and in Congress.
The goal was expansion to at least retain a Senate edge before secession.
Once a Congressional majority was elected in 1860 which would oppose expansion with a President who would not veto any anti-expansion bill, the Confederacy left and then very soon went to conquer the southwest.