In the 1950s, when I was a kid, we would go "over the hill" (i.e., from the Valley to Hollywood or Downtown) and my mother would wear hat and gloves. But in the Valley, everything was much more casual.
I think the biggest problem with the way folks dress today isn't that it is casual, but it is sloppy.
Also, if you look closely at those pictures of folks on the street or at baseball games in the 30s and 40s, they might have been wearing coats, ties, etc. but they were often rumpled and not very clean looking.
I started dressing more casually 25 years ago, when I moved from Colorado to Texas due to the climate. Even so, when it is hot and humid, a pair of light wool dress slacks is a lot more comfortable than a pair of jeans.
Texas governor Bill Clements was a proponent of casual dress. In 1978, he asked that everyone in Texas wear short-sleeved shirts during the summer.