We only say Chag Sameach when we see each other in shul or others in our community. We don't run around greeting everyone with it.
I like to think that I, too, only say, "Merry Christmas" to others in my community.
You appear to have a rather parochial view of your own "community".
I rather prefer to think of "everyone" as part of my community.