It was the North which refused to relinquish the fortification to the proper authorities. That encroachment was the first act of invasion.
I have read articles that assert Lincoln reneged on an agreement regarding the disposition of Union forts in Southern territory. That he had changed his mind and broke an existing agreement.
Even so, that doesn't give them a right to attack the Fort. It wasn't hurting them physically, and if they had just dealt with the hurt pride and embarrassment of it, they would have eventually seceded successfully and without a massive war.
One of the members of the Confederate cabinet begged them not to attack it. He told them that they currently have the support of the vast majority of Northern people, but if they attacked that fort all that benevolence would quickly turn to animosity and hatred.
And it did. The man was prophetic.