Richard Feynman - I think in high school - was offered membership in the exclusive Arrister, but declined it, saying later that essentially all they did in their meetings was to decide, “Who else is good enough to join this fine group that we are?”
It sort of gave him a life-long aversion of joining even scientific associations.
It was the Arista Honor Society and he was a member, though he may very well have withdrawn his membership for the reason you stated, though I suspect his had more to do with his religious persuasion. He was an absolute genius.