Saying you had an illicit affair is....................
He didn’t say that . He said others said it.
It’s also a comedy show (albeit unfunny) so it gets further protection.