That’s not what Miss Manners said!
I think there’s a distinction in this rule between the short, tea-length gloves and three-quarter or opera gloves. Longer gloves, especially those that button up, are such a pain to take off and put on that one is expected to remove them only for meals.