Although your statement is true and a valid caution, it makes me think you know the media version of the Patriot Act, not the real version. No freedoms were lost. The courts were still required for approval of taps, etc. It basically just brought existing laws into the present by accounting for cell phones, the internet, and such.
Nothing ever seems to grow smaller in scope in the world of government and what the act says now could become radically different under future administrations. There is now a foot in the door that is waiting to be pried open by a charismatic Dim in the future.
I prefer to close the door now rather than risk a serious cold later because the draft of incrementalism is always blowing from the left.