I can see 80 pounds making a difference, you're starting to get into the serious fractions of a hundred then, but 50 or less, doubt it. And even at 80 pounds the difference is going to be slight enough to be easily overshadowed by driver skill.
I'm not a physicist or car designer (obviously), but I wonder if the driver's weight is a minimal factor because the cars have so much horsepower. Once the car is at speed, the driver's weight becomes part of the total forward mass, and might be negligible on an oval track where the speed doesn't vary much. But on a road circuit, if there are corners where you slow way down, the driver's weight might be more of a factor because it acts as a drag every time you accelerate. But to tell the truth, I don't know what I'm talking about. lol