If you aren't doing anything wrong, you have nothing to worry about. They wouldn't be listening to you just for the fun of it.
I imagine Big Borther justified the intrusive telescreens in 1984 in much the same way....
Though I agree with you, for now. What happens if the government outlaws something that today is normal and legal? What happens if they outlaw conservatism? Or Christianity. What happens if at home you speak less than approvingly of a gay?
Do you remember what the DA in Fla. did to Rush, looking for any proof that Rush did something wrong? Imagine the government doing that to you, because you didn't toe the party line.
Your at home and say something against Rosie O'Donuts. Another time someone cuts you off in traffic, and in the privacy of your car, which your alone in, you make a comment about the person, who happens to be of a racial persuasion other than white male. Maybe your praying with a friend, and you pray something that somebody somewhere takes offense with.
Now the government comes along, because they don't like your politics or viewpoints, and they subpoena your cell phone records to find something, anything, to attack you on.
Can't happen? Time to wake up and smell the coffee. We're just a few steps away from it.
Erode a little freedom here. Become a little complacent there. Become a little desensitized here and there to societal attacks.