these agents of foreign powers shouldn't be getting US civilian trials - they should be getting military tribunals, or perhaps special non-jury trials in the FISA court. that's where you argument falls apart regarding "everbody's rights being thrown away" because of these foreign intercepts being handled without warrants.
I wonder what the outcome of the Padilla case will be. That case is about military v. civilian detention and military v. civilian prosecution and sentencing.
Wiretaps is a fairly benign event in most people's minds, so it is a weak venue for probing the question of how much power an executive can have. More attention is paid when the stakes are incarceration. The executive argues that this being a time of war, and Congress having passed an "all necesary force" resolution, there needn't be Congressional or Court oversight of executive incarceration of suspected terrorists.
That action wouldn't cause me to lose any rights either, but it's treading in an area that makes me rather nervous.