You seem to want to take a literalist view, until that idea fails to meet your idea of what the Bible says.
There is no indication that the light first created before the Sun came from any source or that source would cause night and day on earth. You seem to postulate a 24 hour blinking light cycle or some other idea that is clearly not Biblical.
I wouldn't be so quick to charge someone else with not believing God's word from now on, if I were you. Just because your pastor told you the Bible says that there were 24 hr days does not mean the Bible really says that. Why do you want to give the morning and evening passage more weight than the creation of the Sun? It is obvious the morning and evening are part of the thought poetry of the Hebrew in Gen 1.
I think I have shown there are far better reasons to say that the Bible shows much deeper and more accurate meanings than most literalist pastors would consider. This is probably because they lack adequate training and are rather simplistic. That is ok. Before most people could read, the Roman Catholic church devised the mass to teach the basics of the Bible without any literacy at all.