The key word here is potential.. The notion is that we shouldn't have more effective counterterrorism and law enforcement because these powers might potentially be abused. Indeed, some people are more concerned about potential abuse from the feds than actual abuse (or far worse) from terrorists. And frankly, that's just plain dumb. It's the kind of thinking that, if taken seriously, will eventually earn us a Darwin award. Fortunately, not too many take it seriously. In particular, people who are actually responsible for national security don't take it seriously -- at least not in this Administration.
Moreover, you are assuming, wrongly, that the Patriot Act somehow abolishes due process or judicial review. It doesn't.
If a law can be abused, it will be abused. Cynicism regarding political matters is robust mental health. Lack of it is gullibility.