He didn't have Grant's Army and logistics then did he? He wouldn't have needed to invade anybody if he had Grant's army and logistics. Nobody would have dared attack him except through dire necessity.
As the North had no necessity to attack the South, the war would have never been began.
In fact, both Lee and Jefferson Davis were committed to offensive military actions because they believed, correctly, that often the best defense is an aggressive offense.
That's why Lee marched on Maryland in 1862 and Pennsylvania in 1863 -- offense would make the Union sit up & take notice like no defensive strategy could.
So,from the beginning Confederates sent armies into Union states & territories -- Maryland, Pennsylvania, Ohio, West Virginia, Kentucky, Missouri, Kansas, Oklahoma & New Mexico.
There was nothing "defensive" about those.
Confederates also claimed Union states & territories as their own.
So the whole idea that Confederates just wanted to be "let alone" is pure propaganda.
Sure, they wanted to be "let alone" -- to consume & destroy as much of the USA as possible.