Do not discount physical activity.
When you earn your living pedaling a pedocab for tourists, you can burn a buttload of calories. Same goes for planting and harvesting crops by hand. Many countries in Asia still use good old-fashioned stoop labour to plant and harvest that rice. No, I am not advocating stoop labour, but I am saying that a good deal of exercise will go a long way to making one lean.
You're absolutely right. I let myself fall into the conventional wisdom that diet is all. I should know better from my own experience.