To a degree that's true, but it avoids the difficult question of the Apostles.
There is a Spiritual lineage that is significant, but there is also an Apostolic lineage that is also significant.
If it is true that the Church would never be defeated, then there are churches that can trace back to the Apostles. I don't think it's the intent for John Q. Christian to just hang out a shingle and declare himself pastor and church.
I agree. God's church (those who have been called out and have received his holy spirit) have generally gravitated to various organizations throughout history.