If it is not in the Constitution, then it is a created right. In this case, it is a right invented by the Supreme Court. The Government giveth and the Government taketh away. If you believe the right to privacy originated in the government, then it can at least be qualified, if not eliminated entirely, by them.
You're not "giving them an inch." They're giving YOU one. At least, according to your theory.
Either you're a visionary or a paranoid shadow-chaser. I guess time will tell which. It will also tell whether I am a patriot or a dupe.