I walked out, have not returned and I have never felt any remorse over it.
Faith is in your heart- not in a building.
I stopped attending my church after 9/11 as well, for a couple o' reasons. However my wife is still Catholic and attends Mass most every Sunday. I was perusing the bulletin last Sunday, and there was a thing about Hiroshima, and how Paul VI (I think) said that was the worst mass destruction event of all time, and praying to end all nuclear weapons, and how the US has the most, and Hiroshima/Nagasaki killed a bunch of people, yadda yadda. It has so many half-truths and inaccuracies, it made me sick. It just reinforced my decision to stop attending Mass there.