You are correct, and I should have made that clear.
The four Upper South states would not secede just because of slavery, but when forced by war to chose sides, they chose the Confederacy.
Which raises the question - we just had a filibuster arguing that the current president cannot do drone strikes on American citizens.
How is Lincoln’s response vs Virginia any different? The state did not take up arms against the federal government, and Lincoln went to war against them anyways.