It just doesn't work out from a numbers standpoint.
In the 70s I could fly myself and a full plane of friends somewhere round trip for quite a bit less then an airline ticket. Roughly speaking, it was always more economical if you went to the range of the tanks on a 182 or 210. Twins not really...but even then, a 310 rented for about $100/hr, and they did go fast. You could pay the premium and have some fun...
But that's not true now. The regionals are cheep cheep cheep, and "upper middle class" people now make far less then they did even in the moribund '70s: who can afford a Piper Archer that costs FIVE HUNDRED THOUSAND DOLLARS for a lousy 180 hp 4 place airplane!?
Cirrus did a great job building a modern upper class airplane and what they got for their trouble was bankruptcy and Chinese ownership.
For a variety of reasons, it just don't work. Perhaps with the drop in gas price things can improve...but the unit costs are just too high to keep GA going.
Cheaper to buy a flight simulator and do it in your living room. Sad that it is that way...
When I was young, I could get flight lessons from WW II, Korean, or Vietnam veteran pilots for a song. Many of the instructors were friends of my Dad. I would wash and refuel planes and could earn some of my lessons and fuel.
Insurance was cheap, and airspace regulations outside of the big airports favored general aviation. There were no FAA rules for expensive avionics or navaids. If I wanted to fly our friend's Champ, Cessna 150, or Taylorcraft, I could do so with minimal equipment on board.
Today, all of that has changed. And as you say, the numbers don't add up anymore.