Because so many are terrified on two types of grounds
1. Their faith is shaky and they do not truly accept their literalist interpretation of multiple translations of the Bible on faith.
2. They are afraid that if the above mentioned literalist interpretations are notcorrect, then humans will have no moral underpinnings.
I think you are certainly close to the core of the issue.
My understanding of the "fear" Biblical literalist have of evolution is the idea of death before Adam's fall and the effect that has on the theology of the entire Bible.
I get the impression from all the various material I've read that a large number of young earthers could convert to old earth creationists and accept much of evolution's timelines (aside from some of the species issues) if death before sin could be explained and accepted Biblically.