Again - the Confederacy did not invade the North. The Confederacy was invaded, and they fought to protect themselves.
This is why you imprisoned the legislatures in Kentucky, Missouri, and in Maryland so that you wouldn’t see a repeat of what happened in Virginia when they refused to take arms against their brothers and sisters.