The Tower of Babel account makes it clear the people tried to build a tower to Heaven. It is true it failed. But that event also explains how we got the various languages around the world.
If Genesis isn't an accurate account of events the rest of the Bible has to be called into question. I think that's why a lot of "scholars" have taken the position you noted.
A cradle Catholic, I was taught that the Bible is divinely inspired and free from substantial moral error, but that it is not a science or history text and should not be taken as such. That leaves a lot of room for everything from simple errors in fact to literary invention. And it is hard for me to believe that written in Old Hebrew script as the essential religious text of a minor, insular people, that the Book of Genesis and the rest of the Old Testament somehow made it to Japan, China, and India, the Americas, and so on as to become the basis for all the world’s flood stories.