It's not outside the realm of possibilities at all, and in fact, I think the idea probably explains the phenomenon quite well. There us a definite "language area" in your brain - "stroke out" the wrong areas and people can speak, but not understand language, or visa versa, or both understanding and speech can be destroyed.
But it's clearly not a physical area of the brain, since many stroke victims do recover the ability to speak.
Scholars and researchers have been beating on this question for the last century, so it's equally interesting that there's no solid answer on it yet.