Being full of Southerners and NOT even states, that was a fair claim to be made. Ever read a divorce settlement?
You contradict yourself. You said that the south “didnt attempt to garrison Northern forts or claim areas outside of the Confederacy” but now you rationalize that it was somehow OK because there were southerners there (there were northerners there too) or that they weren’t states (California was a free state).