There’s something about your body that makes it wildly inefficient. If you need 5000 calories or more to maintain a 120 pound frame, then there is something about your metabolism, digestion, or genetics that makes it that way. You can say that people process foods differently (and they do to some extent) but the truth is that you’re probably an extreme outlier on the scale.
It’s quite simple. Eating a balanced diet (consuming adequate amounts of macronutrients) while managing your calorie intake and energy expenditure can and will explain the weight gain, weight loss, and weight maintenance of the vast majority of the population. Exceptions like you don’t invalidate this, nor would an obese person who eats very little.
My experience has been enough to make me not willing to pick up and cast any stones.
/johnny