Unless you believe the gov't should get to play thought police in our lives it certainly does matter the locale of the comment. If Graham's expression of his religious beliefs disqualifies him to speak at a gov't event, we do not believe in religious freedom in this country at all.
He can say whatever he wants. He doesn't have a right to speak at government events. No one does. There are agendas and invited speakers. No one has a right to be invited.
Your point about where he said it is insignificant. The point is these employees heard about it and asked their employer not to have the guy speak. If christians had done this you'd be leading the charge.