The thing that looks like an "X" is not an "X." It is the Greek letter, Chi (equivalent to our "Ch"), the first letter in Christos, "Christ."
Go into most any historic, traditional Christian church (e.g., Roman Catholic, Lutheran, Episcopalian, Greek or Russian Orthodox), and you will probably find that "X" on the altar or pulpit or somewhere. You'll find it in one of the most ancient and universal symbols of the Christian faith, namely, the Chi Rho, which looks like an overlaid "XP" but actually is an overlaid "CH-R," the first two letters in Christos:
BTW, the other half of "Xmas" usually is ignored, but it too has religious significance. It refers to the "Christ Mass," the Divine Service of Word and Sacrament celebrated on the Festival (i.e., the Feast, referring to the Lord's Supper) of the Nativity of Our Lord.
Finally, even the seemingly innocuous word, "holiday," is religious. "Holiday" = "Holy Day," a day set apart for special religious devotion.
The secularists just cannot do away with the pervasive religious heritage of Western Civilization. They remind me of the Communists trying to expunge every mention of unwanted names from their society.