But how is your freedom in any way compromised by the government having a list of who you called? If they used that information to harrass you, that would effect your freedom. But it they aren't using it to harrass you, what's the harm?
The 4th amendment was not a license to commit crimes in secret. It was protection against being harassed without cause. If your phone calls have a pattern suggesting illegal activity, and those records can be collected without harassing you, that's not a violation of your freedom.
I think my freedom is much more effected by having to submit to searches of my body and effects just to fly from one place to another.