They only listen to what is said after you call their name. Right.
So we know they are listening all the time. The question is, do they keep/search/index anything they hear and perform speech-to-text on? I would bet you if you're phone is connected to WiFi they figure you'll never notice a little extra data/traffic out, and they occasionally send a few packets of compressed information.
This should be easy enough to test. I'll come up with an oddball topic, talk about it within "earshot" of my phone, then see what pops up online... ;-)