If you are aware enough to be able to read English words and understand what they mean then you should be able to muster more of a defense than just, "I don't know how to speak your language."
You've never taught ESL, have you? I have and reading and writing English v. speaking english are totally different skills.