It is your inane position that is insignificant. Nobody is singing about those old cottonfields back home in New Mexico, Utah, or Nebraska. The Congress made it legal to take slaves to the territories and nobody went. The Supreme Court said the Constitution prevented Congress from passing any law preventing the taking of slaves into the territories. Nobody went. Total 1860 slave population in the territories -- 44.
Let me repeat from an earlier post on this thread, from arch-secessionist Robert Barnwell Rhett:
"What has been the policy pursued in Kansas? Has the territory had a fair chance of becoming a Slave State? Has the principle of equal protection to slave property been carried out by the Government there in many of its departments? On the contrary, has not every appliance been used to thwart the South and expel or prohibit her sons from colonizing there.... In our opinion, had the principle of equal protection to Southern men and Southern property been rigorously observed by the General Government, both California and Kansas would undoubtedly have come into the Union as Slave States. The South lost those States for the lack of proper assertion of this great principle....
"New Mexico [Territory], it is asserted, is too barren and arid for Southern occupation or settlement.... Now, New Mexico ... teems with mineral resources.... There is no vocation in the world in which slavery can be more useful and profitable than mining.... [Is] it wise, in our present condition of ignorance of the resources of New Mexico, to jump to the conclusion that the South can have no interest in its territories, and therefore shall waive or abandon her right of colonizing them?"
It is not so important what had occurred up to 1861, but what the southern leadership wanted to eventually happen. There can be no doubt the southern leadership coveted the territories. Rhett realized that these lands were unsuitable for cotton or tobacco growing, but he also envisioned other uses for the slaves.
Also, you probably are unaware the the southern San Joaquin Valley of California become an important cotton-growing part of the state in the first half of the 20th century. Rhett had his eyes on California and the mineral wealth of the southwest.
You might be intereested in these two references:
"Slavery in New Mexico" DeBows Review No. 26, 1859
"The Cotton Fields of Arizona Territory" DeBows Review, No. 24, 1858.
Fehrenbacher, in The Slaveholding Republic, notes that DeBows correctly pointed out that the Gila River area of Arizona and southern California had "potential to support cotton agriculture." The professor also notes that the New Mexico Territorial Legislature passed slave code legislation in 1859, "as preparatory for some larger southern vision."
It certainly seems that the slaveholding interests were making a play for the territories in the southwest. Perhaps they should not have been so keen on secession, which settled the matter of southern slavery expansion once and for all.