God has stated that He prefers tabernacles not made with human hands. His indwelling is all that matters, and the gift of the HS comes from Him
There is a clear Biblical record of it. You just choose to interpret it differently. The Bible is not an exhaustive history of the Church.
Your Elisha analogy fails because we are not aware of any Hebrews claiming this authority through the ages. That is not true of the church. Bishops have been around and recognized since the beginning.