As long as I have had the inner additions that I need, no temp has caused me any problems, the great misery is when your bag and clothes are wet and trying to sleep in the rain at 40 or 50 degrees.
Something that is useful if you misjudge and get a chill during the night, is to be able to reach into your pocket and pull out a pocket warmer to activate inside your bag.
Do you carry a SERE kit that you spend countless hours perfecting and personalizing?
In fact no. When I travel outside the city by car in winter I take a couple of army MREs, a bottle of brendy, a propane stove, sleeping bag and an arctic jumpsuit for emergency. I always have a big knife and mandated automotive first aid kit.