Unless you can accept the Bible sometimes uses metaphoric expressions. When the Bible talks about serpents, I understand it as usually talking about Satan and evil and not a literal snake. The idea that a day must be taken in the literal sense even before the day was invented is kind of odd. Just because the rest of the Bible uses 'day' literally does not completely rule out the possibility that Genesis could have used it metaphorically. It is a good arguement, but not an absolute one.
Two things to consider. First of all, in the Genesis narrative, the author was very careful to use the phrase "and the evening and morning was" the first day, second day etc. The length of the day is defined by the context in which it is written. Otherwise, proponents of the "long day" theory need to explain what it must have meant to the ecology of God's new creation to have a long night for each long day.
Also, confirmation is found in the giving of the ten commandments, where God's creation week is directly linked to the Israelites' workweek and keeping of the Sabbath -
Exodus 20:8 Remember the sabbath day, to keep it holy.
9 Six days shalt thou labour, and do all thy work:
10 But the seventh day is the sabbath of the LORD thy God: in it thou shalt not do any work, thou, nor thy son, nor thy daughter, thy manservant, nor thy maidservant, nor thy cattle, nor thy stranger that is within thy gates:
Exodus 20:11 For in six days the LORD made heaven and earth, the sea, and all that in them is, and rested the seventh day: wherefore the LORD blessed the sabbath day, and hallowed it.
The Israelites were instructed to observe a literal day of rest to represent God's day of rest. Considering how the two days are deliberately related by the commandment, I just don't see how we can interpret one day as literal and the other as figurative.