I was always under the impression he left because Disney made him an offer to star in several movies. Andy Griffith has always alluded to the fact he had "problems" during the later seasons and I have always assumed it was alcohol. The Andy Griffith character is dramatically different in the first two seasons than in the subsequent seasons and in the last few seasons he always appeared downright morose.
You might be right. If you look at him closely in those color episodes, you'll notice that his eyes sometimes appear to be bloodshot.
You're also correct that Andy is different in the color shows--especially after Floyd left. He's no longer the quick-witted country boy. Instead, he's the easily-irritated old man. Painful to watch.