There was also a little something called the "Dred Scott Decision." Northerners had to worry about slavery being imposed on their states against their will. It was a real fear at the time that the Supreme Court would overturn anti-slavery laws.
Given that Article IV, Section II pretty much made all anti-slavery laws pointless, I can see why they would be worried. Their position had no legal leg to stand on.