At my church??
"The church" isn't a building.
God's people (the wheat) compose "the invisible church". Those who are not God's people (the tares - the unregenerate who don't have "the mind of Christ") are the ones who "go" to where the invisible church may be meeting and together they make up the visible church.
The tares are growing unnoticed among the wheat (because outwardly -the way people judge things - they look just like them) -- until Christ returns and exposes and separates them out when he establishes the new heavens and the new earth.