God has provided grace to the wheat that is larger than the problems that tares can cause. Also if a former tare believes, it becomes wheat. They won't believe except by hearing, and won't hear except by the word of God.
In some sense, the holy church is the set of all believers. If you are at a place where you're pretty sure there aren't any or they are so outnumbered and/or so weak that the problems overwhelm, then maybe you should look elsewhere to go. If you can. I'm not a big believer in "churches of 1."
Yet, I would think, that a church of one, is the purest form of the church. A tabernacle, even.
Every shepherd in the bible, and there were many who became prophets and kings, were essentially tabernacles.