True. But also remember humans, and only humans, learned to use both fire and clothing to help keep warm. After those developments perhaps the need for full body hair for warmth diminished. Additionally, it's probably slightly healthier to have less hair, as there would be fewer parasites and things like that. To me it seems natural that we lost fur/hair because we really don't need it like animals do.
Except that hair/fur would protect against the elements better, like frostbite, and sunburn, and skin cancer. With the amount of time and energy people have expended over the years to keep warm, you'd think that hair would be an advantage. More energy could be channeled into feeding and raising young.
If hair kept them warm, they wouldn't need clothing and so would not likely think it up and start to wear it. The need would have only arisen AFTER the loss of hair and the need for other protection.