Unfortunately, most who 'celebrate' Christmas are not celebrating Christ's birth and have little or no desire to know Him.
Granted and if, for that reason, you do not wish to participate then that is your decision. However, the actions of those who do not celebrate Christ's birth do not reflect on those who do.
"Therefore do not let anyone judge you by what you eat or drink, or with regard to a religious festival, a New Moon celebration or a Sabbath day." Colossians 2:16