You are correct. I used to manage the bar at a prominent Country Club. Most of our waiters were legal immigrants from South America or the Philippines. They all spoke English very well.
We had a space where the coffee, tea, bus carts, etc were kept. There was a divider between that space and what we called the ‘Grill Room’ which was the golfers everyday bar.
The waiters used to gather back there and chat in their native language and they were loud. I constantly had to tell them to speak English when they were where they could be heard. What they did outside the clubhouse was their business.
I don’t think they did it on purpose but boy did the members get perturbed over it.
It's rude because many people use it as "code" when gossiping about others within earshot. Even if that's not what they're doing, it raises that suspicion.