If the Southern Patricians had said, your right, slavery is wrong, we henceforth free all our slaves. You think there’d still been a war?
In the presidential election of 1860, the right of the slaveholder to take his slaves—property recognized by the Constitution and laws of the land—into the territories, was an issue made by the Republican party but no question as to slavery where it already existed, was involved.
On the other hand, Lincoln, in his inaugural address on the 4th of March, 1861, expressly declared that he had no authority to interfere with slavery in the States, and no intention of doing so. And not until the promulgation of Lincoln's emancipation proclamation, which went into effect on the 1st of January 1863, made without shadow of right or law, and in direct violation of his solemn declaration and oath of office, was this issue raised, as a war measure, to strengthen the Union cause, which was then on the wane, among the abolitionists at home and abroad.
The New England Yankees, who first imported the negro to America, and who had sold their slaves to the Southern planters, because slave labor was unprofitable at the North, and who had engaged in the African slave trade until this was prohibited by law, at the instigation of the South and against the protest of New England shipping interests which was largely engaged in the African slave trade, and had become rabid abolitionists, now demanded emancipation as the price of their loyalty to the Union cause.
While I was not an original secessionist and voted for the Union candidates for the Convention, yet when the North determined to wage war on the South; when Lincoln called on Virginia for her quota of troops to coerce the seceding States, and when Virginia seceded, it did not take me two seconds to cast my lot with Virginia and the other Southern States. Here I took my stand then, now and forever, and will never give aid in any way to those who were enemies to my State and section, many of whom are still haters and traducers of the Southern people."
From :REMINISCENCES OF THE WAR OF 1861-5
by W.H. Morgan (11th VA Infantry)
They could have said whatever they wanted. He was not going to allow them to leave the union.