Look, I'm pretty fat and pretty lazy. I'm not judging anyone. There are going to be enormously fat couch potatoes in every class and place; the folks who just don't try. I've seen 500-pound men in bib overalls and in tailored suits.
But among those who do try, it's easier for some folks than for others. That's the margin. That's the difference. For me, taking a walk around the block or a bike ride is a simple matter of getting off my flabby butt (and navigating the not-inconsiderable hills around here). For someone in a bad urban neighborhood, it's a matter of dodging muggers, gang-bangers, buses and cabs -- after they get off their flabby butts.
How much more difficult is it -- 10% harder? 20%? 30%? That would more than account for the observed differences. And you'd certainly expect the working poor to be more trim than the desk-bound middle class; Physical labor pays less than mental. When you spend your working day hauling drywall or roof shingles or timber up a 20-foot ladder, you don't worry too much about which gym to join so you can go get a workout after you leave the job.
The urban working poor, often, have the worst of both worlds. They're working a grill, busing tables, washing dishes. They have neither hard physical labor nor good pay. After a long day, they just want to drop.