Keep the sun off your head and to tip to ladies.
Hats always came off in the house.
In any event, a hat worn inside essentially becomes an ornament on one's attire, and causes no significant problems that I'm aware of, so flouting such a social norm seems harmless at worst. After all, some people wear sunglasses indoors for no good reason...
“It’s a free country.”
Yes, it is. But, freedom to do something doesn’t mean it’s the thing to do. After all, half the voting public voted for Hillary. :-)
Except for special occasions, usually solemn, I don't take my hat off indoors unless I'm home. It may have been a custom at one time but those days are long over with. I do see some of the older guys taking off their hats indoors but far more don't.