I know several ultra-wealthy people.
Most people could not bear the amount of stress and effort that they put themselves through for years on end to achieve such. You don't waltz into loads of wealth, it usually requires taking grave financial risks and working 80+ hours a week every week for years on end without vacation and no real hope of financial gain.
Quite frankly most people do not have the fortitude to do what it takes to accrue great wealth in a relatively short amount of time. It isn't hard to do in theory, but it pushes a human being to the breaking point and only the strongest survive in practice. I doubt most people can fathom the amount of determination it requires to carry such a burden without rest for years on end.
For most people, the personal cost, stress, and pain isn't worth the benefit of acquiring all that wealth. It is MUCH easier to coast by on 40-60 hours a week on a regular income. Of course, most of the really successful wealthy people that you meet aren't really in it for the money per se. To them, the money is often just recognition of the monumental effort they put into whatever it is they do. For a lot of them, it is just in their nature to build things and work hard.