I have done that too, with people who tried to speak spanish to me. I always respond in German. They find it irritating, which is the desired effect.
My dad sent me to the store for lettuce a few days before Christmas. As I approached the produce area, a couple women switched from speaking in English to speaking in German. I selected my lettuce, then moved along after saying, "Fröhliche Weinachten". Switching languages as an exclusionary tactic doesn't work well in this country. I passed my German proficiency at UCSD by taking the SAT II test in German, then successfully completing the oral proficiency with a 30 minute discussion of an article from Der Stern with a native speaker. I don't use it much anymore, but I found it very handy while attending school. A couple of the technical books I needed were offered in English, French, German and Russian. English copies were available for a week and all checked out. German copies were available for the whole quarter. I took the German copies.