I just slotted into the middle of it all. I have good gear to protect me from the elements, so the elements aren't too bad. I haven't done anything too extreme personally, though. Maybe 100 km of dirt road at a time and riding all day in the rain on the way home from the east coast - every time I've been down there so far. I'm originally from Halifax/Dartmouth, Nova Scotia, which is why I keep ending up down there. I know guys, and a woman, actually, who have done the trip up to Prudhoe Bay or across the Trans-Labrador highway, but I haven't had the guts to do it myself.
But seriously, if you do any riding in cool conditions at all, an electric vest is great. A lot less bulky than sweater, and more effective. Combine with a light sweatshirt and you're comfy in temps you'd be only tolerably warm in with sweaters and stuff under your jacket. And basic ones, live I've got are cheap, like under $100. And they fold up small enough that you could realistically have it with you all the time. The one I've got draws 60 watts, I think, which all but the weakest electrical systems should be able to handle.
If you ever decide to come up this way, don't worry, even the east coast and somewhat more northerly areas are generally pretty decent in the summer. Stinking hot and sticky around here, though. Whereabouts are you? Sounds like yer down south, maybe?
Kansas City