The people who left there mostly did NOT own the real estate they lived on, so why would those who owned nothing want to go back and be again below sea level and exposed to trouble. Nope, they are going to stay out of the state.
Many of them have found new territories to rob and rape. They don't care where they live as long as somebody else is providing a roof over their heads.