Don Knotts left the show after their last season filmed in B&W. I suspect that is the main reason it went rapidly downhill.
You’re probably right. Even though I wanted to strangle that skinny little idiot, Barney did make the show what it was and Griffith made sure of it. He took a backseat to being the straight man for Knotts character and it worked. Yep, when it went to color all we had was Helen bytchier than ever and Andy groveling more than ever.......LOL