To that I can only repeat:
If you take the time to dine with someone, you must also take the time to converse with that someone - even if it is empty small talk. If you want to read - stay at home in your study, or go to the library.
You claim you decline invitations to large dinners “75%” of the time - the rest you accept to remain “socially viable” - how exactly do you remain viable while keeping your nose in a book at the table?
Just being there is a big part of socializing. Some people can have a great time together without directly talking. Those who invited him likely knew that conversation would be a problem for him, and he may be mentally elsewhere during the event, but still desired at least his presence.
Hey, if most of the 10 people at that table wanted him around and were OK with the situation, fine. Awfully, um, Leftist to recommend social exile due to a situation most of the participants were OK with.