See the graphic immediately above your post: Note how the magnitude of the distance change, and its directions, have changed over recent time.
"Average" distance per year is now very rapid: though I admit I don't know the latest rate exactly.
I learned to fly in 1954, and "magnetic variation", the difference between magnetic north, where the compass needle points, and true north, direction to north pole, was hammered into my head. Back then, in north Louisiana it was 6 degrees east. My most recent USGS map shows 3.5 degrees east for 1995, so it's probably down around 3 degrees now. The compass variation is zero somewhere in the mid-eastern part of the USA and becomes westerly toward the Atlantic coast, with local anomalies due to mountains, ore deposits, etc. So there is definitely a slow, but easily measurable, change.