The reports of W&L students never mingling with VMI cadets is grossly mistaken, at least in my time at VMI.
I never had any problems with Veemies. Some of us sneaked onto the grounds once during Mock Convention. There were some dates/girlfriends present as well (this was before the Asteroid hit). Some cadets invited us into their dorm room and we had a good old time visiting.