I would not make a scene if I had been refused, but it has never come up.
I’m thinking that you go to one in an area without a high homeless population. “Customer only” bathroom policies are quite common.