Rafe was a friend and mentor of Andy. Andy wanted him on the show more but Rafe wanted to concentrate on his singing career.
You're absolutely right. Just found this about Rafe (Jack Prince).
http://www.tv.com/people/jack-prince/
Andy Griffith, who had met Prince when they worked together on Broadway in Destry Rides Again, wanted him to accept a larger role in the series, but Prince declined, believing that it would hurt the singing career he was committed to.In September of 1991, after almost three nondescript decades on the local theater/nightclub circuit, Prince surfaced again to take part in a partial reunion of "Andy Griffith Show" cast members on TNN's Nashville Now. Almost 30 years after his last appearance in Mayberry, Jack "Rafe Hollister" Prince was greeted with thunderous applause by the show's faithful fans. Prince passed away in California three years later.