Which of your rights is eroded by leaving it there?
If I walked into the court to have my case heard, and the building was dominated by a Christian monument and the Chief Justice was proclaiming that we are all governed by a Christian God, it is not that big a deal. I worship that God, as well. But if someone who doesn't represent that God comes before the Court to have his case heard against a Christian opponent, would he not be justified in thinking that the other side was the "home team?"
Our Courts are supposed to be about impartial justice. A court that openly favors the views of PART of our population does not give the appearance of impartiality.