Personally, I think Downey was talking about a real game of ping pong where he smashed the table. But the reaction between him and Letterman tells me there was a double entendre at play.
ThankQ ponygirl & generally, for the info on comet pingpong.
Perhaps Downey’s invitation to that party was his sorry introduction to that disgusting world. I would imagine there were males at that gathering wanting to have a turn with the good-looking 18-year-old.