Do not feel bad. To be honest, I have a struggle with the practice of fasting myself. That is why I keep it for Lent only, the real season of penence. I would rather just simply pray more and do good for others.
As I said, I just do not view Advent as a season of penence, but rather a season of great joy, because the infant King is soon to come.
Advent is a season of sober preparation, looking not only to the Nativity but also to the Second Coming. The Gloria is suppressed in the Mass, and the Church reflects on her longing for the birth/return of the Redeemer. Austerity and self-examination are in order; joy can come later.