IMHO man does not have the authority but The Holy Spirit inside you does have the authority.
John 16:12-15 (NIV) 12 “I have much more to say to you, more than you can now bear. 13 But when he, the Spirit of truth, comes, he will guide you into all the truth. He will not speak on his own; he will speak only what he hears, and he will tell you what is yet to come. 14 He will glorify me because it is from me that he will receive what he will make known to you. 15 All that belongs to the Father is mine. That is why I said the Spirit will receive from me what he will make known to you.”
Another neat analogy is that Bethlehem is in your heart where Our Lord Jesus Christ is born.
As long as Jesus Christ took Scripture literally, and quoted Scripture literally, as He did many, many times, there is absolutely no basis whatsoever for fallen man to allegorize or spiritualize or in any other way change the meaning of the Bible as God wrote it.
Aside from the obvious allegories that God Himself put in His book, fallen man has no authority to read the Bible allegorically. God never gave us permission to do that.