That's the problem. Four three years we've been hearing about people's rights being violated, WITH NO EVIDENCE. If (when) we get hit again, I suspect you'll be screaming about another "failure to connect the dots". We can't have it both ways.
Why do you think that there hasn't been any attacks since September 11?
If the intent is terror amongst the civilian population you certainly don't need a huge conspiracy and technology, a lot of manpower, a lot of explosive or even a lot of planning. There are a thousand different things that could be done a 1 to 4 men.
Why hasn't anything happened? You don't really believe the it had anything to do with the PATRIOT Act, do you?