Once the Castro Government replaced the Batista Government in Cuba, the naval base at Guantanamo Bay became the property of Cuba. Especially since it controlled the entrance to the best natural harbor in Cuba.
It wasn’t just about appearance. Allowing a hostile force to continue to occupy the base would have had real world economic consequences for the people of Cuba, as well as undermining respect for the government of which they had created.
I've covered this before. If you were referring to Morro Castle, you would have similar circumstances. Havana was the major port. Guantanamo bay was not.
Guns of Castle Morrow overlooking Havana Harbor.
Apart from that, we had the Fort by Treaty with a foreign government. In a divorce, people keep the property that was theirs before the marriage.