Despair I think comes from not working on your interior life. That inner dialog, yes dialog with Christ. As a priest told me, when you are so busy talking to him, he does not have time to answer.
There IS a middle ground. There HAS to be.
*Doing good for others FOR THE GLORY OF GOD has to count for something.
*Listening to a BORING neighbor who daily spouts his troubles has to count for something.
*NOT scolding your neighbor for letting her dog POOP on your lawn has to count for something.
*Picking up after the yuppies next door when they SWEAR that they don't drop their trash in front of my house has to count for something.
*Calling a friend whose has HORRIBLE arthritis and listening to her has to count for something.
*Meeting and having coffee with folks that USED to be my friends, listening to all their bodily aches and pains and NOT telling them: "Well, what did you expect with your MEGA-lazy-ass lifestyle?" has to count for something.
LOTS of ways to work on your "inner life" without saying a single prayer.
OK, :)