Yes, it was insensitive of Letterman in a couple of ways. As you say with respect to the gay actors, and again Letterman was even more reprehensible than when Kerry brought up that subject of Cheney's daughter at the debate. It was not fair at all to Dick and his daughter.
Can you imagine if a Republican made those comments?