Roger Williams wrote in 1644: “First, the faithful labors of many witnesses of Jesus Christ, extant to the world, abundantly proving that the church of the Jews under the Old Testament in the type of the church of the Christians under the New Testament in the antitype were both separate from the world; and that when they have opened a gap in the hedge or wall of separation between the garden of the church and the wilderness of the world, God hath ever broke down the wall itself, removed the candlestick, and made His garden a wilderness, as at this day. And that therefore if He will ever please to restore His garden and paradise again, it must of necessity be walled in peculiarly unto Himself from the world, and that all that shall be saved out of the world are to be transplanted out of the wilderness of the world, and added unto His church or garden” (Roger Williams, A Letter to Mr. John Cottons [1643], quoted in Leonard W. Levy, The Establishment Clause: Religion and the First Amendment [1986], p. 184).
Roger Williams founded the Baptist Church.
Madison came later... but was familiar with the materials Williams wrote.