It only matters as a technicality. Over the course of a 500 mile race, I'd guess that driver skill and endurance, mechanical reliability, and a good bit of luck are all much more important factors than a 40-50lb weight advantage (that's the difference between Patrick and an average IRL driver).
As for Robby Gordon, he put his foot in his mouth with his earlier comments, he's been called on it by the fans and media, and now he's having to backtrack.
Hard to argue with you. I used to race formula ford a bit, and the difference of a few pounds is even more important there, and the truth is it's not that big a deal.
My only reason for suggesting it is to eliminate the whine babies. We have rules for so many other things to make the cars equal, no big deal adding the driver to the equation.