Just so we're clear on this point: no law of nature or humans restricted slavery to cotton plantations.
Slaves could be, and were, used anywhere slavery was tolerated.
There were slaves in Deleware up till the 1860s. There were slaves in a lot of Northern states, but the great body of people in the North were not concerned so much with these small bits of slavery here and there, they objected mostly to the large scale plantation farming that employed thousands of slaves.
They would have tolerated this small scale slavery in the territories just as much as they tolerated it in their own states, were it not for the hyped propaganda which had them believing new plantations would be opening up in the territories.
It appears more and more that the whole "expansion of slavery" argument was just histrionic astroturf.