Matt and Trey knew that he had a mild stroke a few years back...they said his mind wasn’t clear and that he was being manipulated...i think they were hurt by the situation and held no animosity toward Isaac.
The episode where they turned Chef into a child molester seemed to show quite a bit of animosity towards Isaac Hayes.