There is a distinction that many refuse to make and that is deliberately blurred by those who wish to render Christianity moot.
That is that people must believe that every word of the Bible be interpreted literally. They make it an either or to try to trap Christians.
Every word of the Bible is true as God is truth. That doesn't mean that every word must be interpreted literally, as there are songs, poetry, proverbs, etc.
The term *literally true* is used to equate literalism with truth and they are not the same thing.
As to why Genesis be read as factual is because it is clearly a narrative account.
The other issue is that many would change the interpretation from allegorical to *literal* part way through the book, after the Flood, for no other reason than their preconceived notions of what must be true because of *scientific* evidence. There is nothing in the style or reading of the book that would indicate that after the Flood, Genesis should be read as factual as opposed to allegorical.
That is totally inconsistent with any kind of grammatical style.
To be consistent, either the whole book be read as factual, or the whole book as allegorical.
Ref:
Norman Geisler, “Conviction without Compromise”, Chapter 17
What is and ISN’T, “literal interpretation”.
What “literal interpretation” does not mean, and what it does not exclude.