I am a Hebrew linguist. Gen 2:4 refers back to all the periods (yoms) of Gen 1. I have studied these passages in the Hebrew for almost 20 years. I don't think they can be read any more carefully than that.
Reading in the English alone, just doesn't allow for good interpretation.
Then you should agree with me, that the word YOM in Genesis 1 which includes the modifiers evening and morning means a literal 24 hour day...which is what this is all about.