My iPhone has a GPS setting that makes the apps ask for permission to use my location which I use.
That helps... I keep mine turned off all the time because like you say... it could still be accessed without you knowing. I just pull up an online map and use that long lost skill of reading maps.
I will admit though that I have “temporarily” turned it on for a few very handy temporary apps and then turned it off and uninstalled the app when done. Like using a satellite finder/aimer app for setting up a dish. Very handy tool if you have ever done this.
Unfortunately certain elements of basic geographic tower pinging always remain on and there is not much to be done about this. It has been a mobile communications standard to comply with the 911 emergency system for many years now. But it is not at all like GPS.