"The fiasco involving Colin Baker" had to do with the then BBC Head of Drama, Michael Grade, who not only admitted he didn't like Doctor Who, but killed other sci-fi shows like The Tripods and reruns of the original Star Trek that didn't cost the Beeb a penny to produce!
Further, after an 18-month hiatus, and "Trial of a Time Lord," it was decided by Grade and another working for Grade who had an equal distaste for the show that Colin would be removed from the show because three years was enough as far as they were concerned, even if the show wasn't actually on the telly.
For the final two seasons, JN-T was only a reluctant producer as he actually wanted to move on but he wasn't allowed to. Having moved the show from its traditional time slot of Saturday afternoons to compete with long-running programs like Coronation Street is what ultimately killed the show. Don't place blame at the feet of JN-T for things beyond his control. He did the best he could with what he had.
I thought that his entire tenure on the program was fairly awful. Too many derivative programs, too many poor casting decisions (some of the most unpopular companions in the original run of the series dated from the JNT regime). Don’t get started on his fights with Eric Saward and others on the series.