Thomas Jackson owned as many as 10 slaves at points during his adult life. Are you saying that Jackson and men like him would look at the territories and say, "Well since cotton won't grow there then that nowhere I would want to live?"
Not all slaves worked the fields. Or weren't you aware of that.
Vast bulk of them did. It is irrational to think the tail wags the dog.
In 1840 about half of all US slaves lived in the Deep Cotton South, by 1860 it was almost 60%.
But that still left 40%, about 1.5 million slaves, in non-cotton Upper & Border South states.
Even in cotton states a fair guess would be only 80% of slaves worked cotton, meaning, overall roughly half of US slaves worked cotton.
Point is: cotton was unnecessary to support millions of slaves.