I do believe it is not just government workers. How about Thanksgiving Day? Easter is actually a more important day to Christians but it isn't a holiday. The government has the power to grant whatever holidays it wants. The government is an employer and can grant days off as it desires. Just because the government grants a holiday doesn't mean that it is because it is a religious day to be observed. You may argue that point but it is true.
In my opinion the Christmas holiday is simply a recognition of the religious beliefs of the general citizenry, and that refusal to accept Christian displays is inconsistent with that. Likewise the display of menorahs etc. should be considered simply signs of respect for the Jews. If blacks want to celebrate Kwanzah, more power to them. But a true recognition of diversity in this country must proportionately acknowledge the religious views of the Americans people. Symbols matter, and minorities should not object to the public display of symbols that are important to the majority.