Nope. Nothing moral or ammoral about it. It's simply good manners.
Take your hat off inside (unless it's part of a uniform or a safety/hard hat issue...)
Take your hat off when the National Anthem is played (many of the same who wear their ball caps in restaurants do not take them off for the Star Spangled Banner.)
Take your hat off when the flag passes by.
It's a matter of manners. Something America could use a lot more of these days.