Makes sense if that was his motivation to not officially join. Sure he has been invited over the years, as was Ron Wood.
I don’t know if it was a him decision or a Stones decision. One of those things nobody has bothered to explain. He’s just been their session and tour guy for 26 years, and apparently everybody is OK with it that way.