Or lie and say yes. You're not under oath. And even if they could prove you were lying, all you would be doing is trespassing which is easily solvable by leaving.
You could say yes and that is a good response, but I like the idea of not accepting that their line of questioning is acceptable. If you say yes, you are conditioning them to keep doing it to other people. Boundaries.