I think hitting a baseball is the hardset thing to do in sports. Hit a round object, with a round implement, and hit it squarely, and it is thrown at you at ninety miles an hour from sixty feet away, and with movement on it (curve, forkball, knuckler, split-finger). The batter has milliseconds to decide ball / strike / fastball / curveball / scroogie and swing or not.
You have a lot of sympathy from sports writers on that opinion. Overall it probably is the most difficult thing to do in sports because of the timing and second guessing involved. Mechanically, though, a successful golf swing is more difficult.