Good for you, but many people also eat far more calories than required without getting fat. On the other hand, chronic dieters may get to a point where they eat very little without losing weight.
Metabolism differs by individuals and by their actions. There are differences by the proportions of fat, protein and carbs eaten. They’re even finding that viruses can affect weight.
And hormones—yikes! Estrogen, for example, encourages weight gain, and we know that various pollutants in our environment and food supply can both mimic and lead to high estrogen levels.
Yes, metabolism impacts calories burned. It doesn't exempt you from the (energy in)-(energy out) equation.