Dairy allergies in your family otherwise?
None.
Billions of children drink milk; I seriously doubt milk has anything to do with it. I’m skeptical of any diet relationship.
Perhaps you are right. It might be the things they are feeding the cows that turns up in the milk. I take a dim view on the quality of American food these days.