I like to think that I, too, only say, "Merry Christmas" to others in my community.
You appear to have a rather parochial view of your own "community".
I rather prefer to think of "everyone" as part of my community.
We Jews have learned over time, through many painful experiences, that not everyone is part of our community and some of those who are not part of our faith would hope for it's destruction.