“Seems to me the city lost the right to make such a claim when they issued permits to modify it.”
You are assuming, probably wrongly, that permits were issued.
“Rightly”