Speculating on the crimes that Letterman may have committed at least give people something to talk about beyond "well, I guess that blackmailer will sure go to jail. Yep. That's fer sure. I agree. He sure broke the law that time, didn't he? Yep. Yep. Sure did..."
Well, there come times when giving people something to talk about is inappropriate enough. This seems to be one of those times, until or unless any of Letterman's former paramours come forward and, rather than saying they were his former paramours (I think two and maybe a third have done so, though I could be wrong about the third), even one of them says he did indeed do something along the line of actual sexual harassment and files a legal complaint against him---as another poster notes, can't have a plaintiff without a complaint.
Until there's a bona fide complaint with bona fide evidence to back it up, speculation on any crime anybody merely thinks David Letterman committed gives nothing but a scratch on the prurient itch, which says manifestly more against the speculators than against Letterman.
You might as well ask the ocean to stop being wet. People will talk and gossip and ain't nothin' gonna stop it.
It's also covered under that quickly-vanishing "free speech" thingy.
It doesn't bother me anywhere near as much as it seems to bother you for some reason.