DiogenesLamp: "The difference is, one was defending their land, the other was leaving their own to go attack others."
In fact, by war's end Confederate forces had invaded or operated as guerillas in most Union states, including:
Maryland, Pennsylvania, western Virginia, Ohio, Indiana, Kentucky, Missouri, Kansas, Oklahoma, New Mexico, Colorado, California and even Vermont.
The Confederate Navy also operated a fleet of commerce raiders (including CSS Sumter, Florida, Alabama, and Shenandoah) which captured many dozens of Union ships.
When any of that equates to sending 35,000 men to conquer your nation’s capital, you let me know.