My dearly departed Grandmother used the phrase to describe those who spent portions of their lives smoking weed or in general using drugs. “Gone to pot”, or “on the pot” were equally interchangeable, with her admonitions.
Hat-trick, your screen name is only two letters short of being an anagram of Tax-Chick.
My late grandmother would say, “Get off the pot,” but I assumed that had the meaning you would expect in a place with only one bathroom that didn’t have running water until the 1970s.
Several sources mentioned an association with marijuana usage, but the “dead or otherwise in decline” usage is older.