Other than certain positive legal rights established in law and not intuitively obvious as a matter of course (right to trial by jury, for example), Federal and State Constitutions do not "guarantee rights".
Constitutions properly should be thought of as restricting the power of governments to those powers enumerated within them that the people have chosen to have handled by their government.
In other words, a Constitution is a document outlining what powers the government shall have among a free people, not what rights a people retain in the face of a levithan state.
The First Amendment does not guarantee the right of free speech, which is a natural human right. Rather, it says that Congress has no power or competency to pass laws restricting speech because the people have not given their legislature such authority.
Since a Constitution is an instrument of the people, and the people determine who shall serve them in public office, it is obvious that the people should be allowed to determine what the qualifications are for those public servants.
An test prohibiting atheists in public office is no more discriminatory than the clauses restricting the Presidency to natural born citizens, restricting other offices only to citizens of the country, or clauses establishing ages well above that of legal majority to attain to high office. The people have determined that they wish to be served in public office by citizens with enough age to have matured to some wisdom. A related qualification stating that such public servants must believe in God differs little in substance, since in essence it is a qualification requiring people to have a rational view of the reality of creation and divine governance of the affairs of mankind, and a future state of rewards and punishments for all men. In this sense, it is like requiring a oath upon taking office or providing testimony in Court. An atheist is obviously incapable of taking such an oath since they cannot make such a promise to a deity they do not believe in.
Furthermore, serving in public office is not a right, but a privilege. As a privilege, it is very rationally subjected to reasonable qualifcations.
Scripture tells us atheists are fools. Can a free people not deny public office to fools?
Equal protection is the law of the land. Deal with it.