Personally, I rent a Piper Warrior when I need the speed for travel to a specific destination.
For enjoyment, I like things low and slow! My home built airplane is the Challenger II. Maximum speed is 100 and a stall speed of only 18. One of the very few aircraft with a 4:1 speed envelope.
This is the difference between a dirt-bike and a Harley motorcycle. Each type of motorcycle is designed for a specific usage. A Harley, when driven on a dirt or gravel road, is down right scary!
Identify the type of flying that you enjoy. If your primary purpose if to fly long distances, then get the fastest aircraft that you can afford.
If you just enjoy flying for the sake of flying, think about something that performs like a WWI fighter. Those are fun airplanes to fly, but often painful for long distances.
For me, I chose the Challenger II because of it's gliding abilities. Shut down the engine and ride the thermals for a few hours. When tired, fire up that engine and fly home.
With a stall speed of only 18, can you imagine the short fields that I can land in? There is no such thing as not being able to find an emergency landing spot. Give me a little headwind, and I could land in someone's driveway.
At the company that I work for, we have a Piper Navajo and two Piper Saratoga for aerial photography. For this application, we need speed.
Identify what you enjoy most about flying first!