Depends on the track.
Having known a few of the top level Off-Road racers and having sat shotgun with a few of them in actual races in Baja, I can tell you that many are every bit as athletic as some Baseball players. The absolute pounding the body takes is something that has to be experienced. My non athletic self had to be helped out of the truck at 250 miles. Took a few days to even want to move again.
Hey, if you think it’s a bunch of fat guys cruising with one hand on the wheel, so be it. If you saw the post I made above I mentioned a series a mag did with Ivan Stewart. As another part of it, they sent him up with a figher jock in a jet and did the routine. Then the jock got in the truck with Stewart and they went through that routine.
Unless you think a fighter pilot isn’t an athlete, the man said it was one of the most intense and physically grueling things he’d ever experienced.
I recall back when CART was still around they did some testing on the drivers, they essentially had very elevated heart rates for most of the race. As others have noted high g-loads, heat, noise etc take their toll. I doubt there is much difference in NASCAR, in fact it may be worse wrestling those big sedans around.
Some baseball players? I’d say 99%. Baseball players have to use bursts of strength and speed, but half of the game is spent sitting in a shaded dugout. The other 48% is spent standing around waiting for a ball to come your way. I’m NOT saying there’s no athleticism involved, but it doesn’t require much stamina. The stamina is required from the fans who have to sit and wait and wait and wait for something interesting to happen.