You contradict yourself. You said that the south “didn’t attempt to garrison Northern forts or claim areas outside of the Confederacy” but now you rationalize that it was somehow OK because there were southerners there (there were northerners there too) or that they weren’t states (California was a free state).
California wasn’t the property of the federal government. It was a free state free to choose.
Any overtures from the CSA were not to take California (or any other territory) as booty, but to allow the inhabitants the authority to choose for themselves.
Figured out the issue of West Virginia yet?