Yeah, if the opposite applied, Eskimos would be GIANTS wouldn’t they?
Israel, in a warm climate, has the shortest average height.
Evolution is hard to predict. It has, on islands, selected both for small elephants, and large rats. First, the variation has to appear, and then the selection has to perfer a given variation. We don’t know if cold weather would select larger mass/surface area or smaller area so that warm winter coats (or burrows) can be provided.