This is not true. I've worn glasses for several years now and if I take them off, my vision is much worse than it was before I started wearing glasses, not just in contrast; I now see nothing but blurs. I don't believe the glasses are "weakening" my eyes either; it's just that my eyes have gotten used to seeing with them rather than straining to see without them. It's certainly more than just the contrast though; there is a definite change in my non-corrected vision.
Has the eye doctor determined a change?