If the building inspector said to remove the sign, I would ask him/her for the ordinance which prohibits ANY signs.
Then, I would ask the city to provide the ordinance that also prohibits such signs.
There has to be something in writing to support either the 2nd Amendment supporter or the city, it can’t be both ways.
Keep eyes peeled for campaign signs in violation of this rule as November approaches.