There was literally no such thing as casual dress in those days. It was also not uncommon for a man to only own one suit. He would wear the same suit day after day for years, having it repaired as needed and sometimes even washed.
When at the beach, men had the option of renting a "bathing suit" at a bathhouse. That was probably the only time in a man's life when he was seen in public without a full dress suit.
People did NOT have as many clothes back then, as most people have today. Even in the 1920s, closets in very expensive apartments and houses were smallish; though there tended to be somewhat more of them, than in less expensive homes. There were cedar closets too, because people had summer clothes and winter clothes, which were moved in and out of these different kinds of closets, depending upon the season.