Yes it is rude, it communicates to others that they are not worthy of your attention, regardless of your motives for whipping out reading material.
It isn’t rude per se. It’s rude if any of the company you’re with is likely to take offense. For example, if I’m eating with my brothers, it’s fine. If I were to eat with someone such as yourself, it’s a safe bet it’d be taken as an insult. So I probably wouldn’t do it in those circumstances.