"You can say that, but I've heard plenty of people (both creationists and not) say that yom always refers to a literal day everywhere else in the bible."
As you probably know, using terms like always, always makes the statement wrong.
It is simply not true that yom always means literal day. You can go to Brown Driver Briggs and see that for yourself, if you like.
Let me ask you this. Wouldn't the yom be plural in Gen 2:4 if it meant a series of seven 24 hr days? Wouldn't it have said, "in the days" refering back to the seven days of creation? It does not
Strong's and KJV advocates insist that day with a cardinal number always refers to a 24 hour period, but this is more a pre-emption of good interpretation that it is a reality.
Since God didn't make the Sun until the fourth day, I hardly see how the concept of 24 hours of day and night is a reasonable concept for at least the first 3 days. So, if it can be a period of time the first few days, it seems it can also be a period of time the next. (I also hear the creationists arguments that light was a different speed at first and other things were speeded up or slowed down. Can you demonstrate a 24 hr day 15 billion years ago or 6000 years ago?)
Internally consistent interpretation demands accepting my view of these passages. To do otherwise is to make a farce out of God's Word.