One of Madison’s main arguments against a tax in support of all Christian religions in Virginia was that state support of religion endangered the state. He stated this idea in 1785 during the Virginia church/state controversy in his epic “Memorial and Remonstrance,***” which set forth the principles and reasons behind the wisdom of keeping state and church separate.
Madison wrote, “What influences in fact have ecclesiastical establishments had on civil society? In some instances they have been seen to erect a spiritual tyranny on the ruins of civil authority; in many instances they have been seen upholding the thrones of political tyranny; in no instances have they been seen the guardians of the liberty of the people.”
"Only a two-way wall, one that both keeps government out of religion and religion out of government, can truly protect religion. If some religious groups are allowed to legislate religious morality or use taxes for church activities, then religious groups too small to influence the legislature will be forced either to live according to the dictates of another religion, and/or to pay money to support activities of other religions. "
"Either way, these minority religious groups will have lost freedoms."
As one constitutional scholar put it: “The wall of separation ensures the government’s freedom from religion and the individual’s freedom of religion. The second probably cannot flourish without the first.”
***James Madison, “Memorial and Remonstrance” (1785), reprinted in Edwin S. Gaustaud, Faith of Our Fathers (1987), appendix A.