"..., that I do not consider the proceedings of Virginia in 98-99 as countenancing the doctrine that a state may at will secede from its Constitutional compact with the other States. A rightful secession requires the consent of the others, or an abuse of the compact, absolving the seceding party from the obligations imposed by it." - James Madison, 1832
I'm pretty sure that Madison read the Constitution, and the Constitution in 1832 was identical to the Constitution in 1860. So obviously people of the time both read the Constitution and believed the Southern acts of unilateral secession to be wrong.
Gosh - I can't find that in the US Constitution. Care to provide a somewhat more specific reference?