Not true, at least in terms of directional sense. My husband never remembers the way he’s gotten to any new place. I remember how to drive to a place I only visited once decades ago.
My mother is still with us in her mid nineties and no longer drives. She can give directions quite well in relative terms, but, as ever, she is poor at compass directions or distances.
You and I are the same in that regard. It’s almost like I have an inner compass in my head.