The problem is that what's held forth as the literal interpretation of Genesis conflicts with what is actually said in Genesis. If "day" (the Hebrew "yom") must always be translated as a single 24 hour period, you've got a big conflict between Genesis 1 and Genesis 2.
Genesis 2:4 refers to the day that God made the Heavens and the Earth, but in Genesis 1:8 God creates Heaven on the second day, and in Genesis 1:10, God creates Earth on the third day -- that is, on two separate days. The Hebrew "yom" in Genesis 2:4 is singular, so by a literal 24 hour interpretation, that would contradict Genesis 1:8 and 1:10.
For Genesis not to contradict itself, "day" can't exclusively mean a literal 24 hour day.
Bah. I hardly concern myself with whether in certain instances the term "day" meant a general period of time or twenty-four hours exactly... just like I don't believe that the "Lamb of God" could have been sheared for wool.
But consider...
1:11 Grasses and herbs are created - day 3
1:16 The sun is created - day 4
1:21 Sea creatures and birds created - day 5
1:24 Land animals and man created - day 6
Plants before the possibility of photosynthesis? There's no way that evolution would allow that ordering. Flying animals before land animals? We're supposed to believe that land-roming reptiles came between fish and birds.
It would be hard to argue that even the order of creation wasn't meant to be taken literally. There was no moral to that part of the story...
I get a charge out of you Genesis critics. On one hand, you condemn the "literalists." On the other, you are incapable of recognizing figurative language when it bites you. You can't have it both ways unless you reject the law of noncontradiction and the logic that goes with it. So which would you rather be, wrong or illogical?