It was fear of death that brought down the Numenoreans.
I guess they didn't believe in heaven...or maybe theirs was a "Valhalla" type of afterlife.
The only people who didn't have to worry about death were the immortals. THERE CAN BE ONLY ONE.