Getting outside doing yardwork is part of my exercise program. That's why I kill myself doing it. Gardening and yardwork are brutal exercise if you do it right, anyway, as you well know. LOL!
I don't do annuals as a rule. I just keep planting various perrennials, it's less effort, especially now that I've figured out what the deer eat.
I do the same as you do...every year more perenials. Of course winter gets them now and then so I have to keep planting those too.
I don't mind doing the window boxes so much, as I don't have to bend over...but the low gardens are getting to both of us.