Didn't stop him in 1862 and 1863.
He didn't have Grant's Army and logistics then did he? He wouldn't have needed to invade anybody if he had Grant's army and logistics. Nobody would have dared attack him except through dire necessity.
As the North had no necessity to attack the South, the war would have never been began.