The day is coming soon when the Spanish speakers will be chewing you out saying,"You live here, you need to learn the language!"
That's why I'm teaching my children to habla Espanol.
That day is already here. I live in LA, and I am often told, "What?! You live in LA and don't speak Spanish?!"
I remember when Spanish-speaking people, when asking for directions or the time of day or whatever, would politely ask if I spoke Spanish. Now, store clerks, people on the street, etc., will walk up to me and start rattling off in Spanish, assuming that of course I speak their language. When I tell them I don't speak it, they act annoyed and put out.
That day arrived here, years ago. I have learned to ignore it, and continue the conversation in English.