Unilateral secession itself was a hostile act. Compound that with the theft of federal property and the assault on a federal fort and you have an act of war.
Double-down by declaring war as the south did on May 6, 1861 and there is no doubt who the aggressor was. That the United States responded doesn’t change that fact.
You are starting to sound like Hitler justifying his attack against those aggressive Poles.