It’s a tradeoff and the financial picture is only one factor. What do you want to use your dollars for?
Part of the financial picture is home maintenance. Around here (metro Cleveland) stats say a family making $35,000 can afford a median house. But how the heck will they ever keep up the plumbing, heating, and other repairs? I live in a relatively small, very well built house, and figure repairs and upgrades average about $3000 a year. Another factor is availability of a good repair support system. Around here, that does exist, but where I used to live in MA that was a real issue.