I recall times decades ago when a mixed group of friends would go to a nice restaurant or a club, one woman would excuse herself and another one or two would rise and offer to join her, sometimes every female in a party of say eight people would disappear at once. I sometimes got the impression that after two went together it was necessary for all the others to go, just to avoid being the one talked about. "OMG did you see her hair..."
We fellows were always left somewhat non-plussed. "I wonder what they're talking about?" "Dunno. Us?" "Uh-oh."
If a group is seated ‘banquette’ in a restaurant, I can see some deciding to go together to avoid disturbing others too much.
I’ve just never seen it to be the general habit that men seem to think it is when they’re discussing female behavior.