Hell, I do this with other white people who are not as articulate. And I don't do it with any persons, including blacks, who are well-spoken. It's just polite behavior. The level of language competence can be assessed fairly quickly when you meet someone. Until you are sure, use mid-grade vocabulary and avoid idioms.
One of my pet peeves is seeing Americans with no foreign language capacity (unlike Europeans and many educated people from Africa and Asia) yelling at foreign travelers in the U.S—newsflash: yelling doesn't make language any more understandable. I speak several languages and have actually apologized to some foreign travelers who were only trying to order lunch and got yelled at by the American bozo behind the counter.
Good point. One generally speaks the language of the listener. You do not want to speak down to or above the average level of understanding to anyone.