Basic thermodynamics at work here....the bigger the difference between your body’s core temperature and room temp, the greater the rate of heat transfer from your body...the more energy your body has to expend to maintain your core temp....the more calories your body has to burn.
Same reason drinking one quart of ice water burns at least 68 calories.
True, before the invention of clothing.
Insulation modifies the equations pretty dramatically.
But really cold weather, you're right. I've camped out in Rocky Mountain winters for as long as a week. 7000 calories a day and still lost 1 to 2 pounds a day.