That's funny. Me too.
I never knew until 1) I went away to college and 2) My dad got a new (really good!) job and moved the family away from the town I grew up in. These both happened at about the same time.
Eye opening, to be sure. It wasn't that I was stigmatized for being poor, it was that everyone else that I grew up with was in the same exact boat, so we never knew anything different.
I only had the one teacher make anything of the way we lived, my mother set her straight pretty quick. Yes all of our neighbors lived the same as we did. I knew people in town had all the modern things- my brother and I would walk a couple miles every Saturday (afternoon- we had too many chores in the morning to go watch cartoons) we could to a neighbor that had a TV so we could watch Science Fiction movies and later Star Trek.
College was pretty eye opening to me too! I worked my way through and found out work in town was pretty easy compared to what I grew up doing. Hearing how others grew up was amazing. I chose to continue to live in the country even when I was working in town. It was pretty startling to my husband when we started dating that I had a house rented 17 miles from the city I worked. To me that was pretty close to town.