Certain types of lavender can be grown in zone 5. I know because I grow them. :)
Zone 4, but it is really a microclimate (4a) because we are in the bottom of a valley. We have drying westerly winds and can have -35, without windchill, some winters. There is a lot of water on the property, but we can have droughts that are just severe enough to preclude lavender.
I have tried it. Fair as a annual and impossible as a perennial. I am fairly experienced with container gardening and an established lavender plant is one of my present goals. I especially want one of the very fragrant ones with more sweet, flowery notes. Some species tend to a more antiseptic odor.