"I was told that the bus driver in question had been reassigned before, and had trouble when he was the driver of the "special-education" bus"
The bus driver is a known problem. Report him.
I agree if he has run Special Ed before and had problems he has issues with handicapped kids.
He's in the wrong line of work.