"A three-yr-old horse is too young to run with weight on his back for these distances..."
No offense, but 132 years of Kentucky Derbys says they can.
And, still speaking generally, racing T-breds are so flaky that they make bad saddle horses. It's hard to get them to calm down and they are more unpredictable than other breeds. And the trotters and pacers are good for practically nothing--try riding one and keeping your teeth. Not much call for pulling buggies--carriage horses in cities are usually the draft breeds.