Not always a few. In the Roman empire, they were about ten percent of the population around the time of Christ. The Temple was the largest public building in the Roman world. The revolt in the early 2nd Century came close to toppling Roman rule in the eastern Med.
Skeptical.