Not really. Perhaps if he had a contested claim of ownership to that toilet, but even then the problem is not in his desire to access it but rather in the manner he attempts to do so. Showing up at 2AM with a shotgun on the shoulder and an open threat to break the door down would effectively preclude his right to access that toilet at the time, presuming it was even his to begin with, simply because the manner in which he desires to access it poses an immediate and real physical threat to both my person and my property.
Now perhaps if he rang the doorbell at 2 PM and politely asked to access the toilet I would have no problem letting him in. Or if he said "let's meet over dinner and resolve our dispute of the ownership over this toilet" I might be inclined to let him access it, or negotiate its transfer at a fair price, or whatever other possible compromises we could come up with. But showing up in the middle of the night with a shotgun and a threat simply doesn't cut it.
By the constitution, those forts were federal territory.
Wrong. The forts in Charleston fell into two categories under the law. In one was Sumter, in the other was everything else (Moultrie, Pinckney, and Johnson). The other 3 forts were all pre-revolutionary and revolutionary era structures that had been built by the British government, the colonial government of South Carolina, or the state government of South Carolina. The use of them was CONDITIONALLY placed under the care of the federal army around 1805 under an act of Congress, passed in the 1790's, that allowed states to do so. South Carolina built and paid for those forts, set the terms of their use, and accordingly had a strong legal claim to resuming control over them.
Sumter differed because it was built by the federal government, though in a location ceded by the state of South Carolina. Though this made it federal property, the cession was itself unilateral by an act of the South Carolina government which presumably could be repealed. The South Carolina government repealed all prior ties to the feds in their secession ordinance in 1861, which would have presumably included the Sumter cession. It should be noted that this is a weaker legal argument than with the other three forts, but nevertheless one of some soundness to it. That effectively made Sumter's property status "contested."