Not for the purposes of the first amendment in general, but for the purposes of the establishment clause. He has acknowledged that Hindus and Buddhists have a God-given right to practice their faith. That's the free-exercise clause.
Under the establishment clause, he's saying that it refers to the establishment of particular sects. In other words, he's saying it's not possible to "establish" Christianity in general at the expense of other religious traditions, because Christianity is not a religious organization; it's a point of view. If you want to disagree with him, that's fine, but it doesn't make him a fanatic. It's well within the views of many at the time of the founding.