No, it isn’t silly. Pasteurized milk is one of the big players in “type II” diabetes, and in Crone’s disease.
I heard it also causes sun spots. I read it on the internet so it must be true.