that is perhaps why the Army got rid of the saucer cap and went to the beret. A beret will fit under an epaulet.
The garrison cap was a sad ‘dress’ version of the WWI field cap, designed to go under a helmet and keep your ears warm. Rather sensible in northern Europe, less so in the deserts of the middle east.
That's a bit of it.
I have a fedora that I really like to wear. Most places have no place to hang your hat like a civilized human being.