No Whole Foods sells to people that have more money than sense. There are two Whole Foods in my area, and yes in more affluent areas. However their selection leaves much to be desired. As for Wal-Mart, I don't shop for groceries there much. What's popular in this area is Lowes Food (competitor of Harris Teeter market). Food Lions are still around but Winn Dixies have left the state completely
Whole Foods actually has "Large number of college-educated residents" as one of its criteria for building a store:
http://www.wholefoodsmarket.com/realestate/index.html