I imagine he hasn’t. So I’m going to say he has NEVER removed someone from a committee for having a too liberal a voting record and challenge someone to prove me wrong.
Well, if you don't have the facts (you're only imagining) I don't know why you would say he has never done so.
I'm sure as Speaker he has assigned and re-assigned because this is what Speakers do every term. Whether they may have been liberal, conservative or just plain inept I do not know.