That is a good point and one the libs would never want to concede unless it fit one of their pet agendas. Even then they would say it only applied to that.
The counter-argument might be that we have delegated that authority to the FedGov. For now. I would not use the language of 'delegating the right to the gov' though because the right remains the people's. We only cede them some authority.
We don’t so much delegate that authority to the government as allow them to exercise it on our behalf.
If you hire a night watchman to watch your stuff so you can sleep, it doesn’t mean you can’t yourself investigate mysterious bumps in the night.