Could you not have a problem with some other component of milk? I think a true allergy would be a reaction to a protein, but I don’t think the issues would be digestive, but rather more like hay fever type symptoms.
Lots of food allergies involve digestive problems. One of my cats developed an intolerance to grain after having had no problem with the grain in ordinary dry cat food for over 10 years. He lost more than half his weight and had stools of yogurt consistency. Within less than 2 days of switching to a grain-free food (has potato replacing the grain) he had perfectly normal stools again.
This is common in people as well, e.g. celiac disease and similar conditions.
My 15-year-old son gets sinus symptoms when he drinks milk. He's been off milk for about 6 months now and doesn't need any of his allergy meds anymore.