Don't be. God knows what's in your heart and your reasons for doing what you did - and He is your Judge; not that pastor.
If others ask you about what happened, all you can do is state your convictions and opinions quietly and clearly. You've done what you could do to show your disagreement with his opinions, and need do nothing further to justify yourself to any man.
During the Vietnam war era, I was stationed in Toronto, Ontario and went to the Anglican church in my neighborhood. Talk about a politcally motivated pastor! He was hell bent on trashing the United States and its' war of agression. I heard that once and walked out immediately. He, the pastor, came to my house and wanted to know why. Of course at the time I put more money in the collection plate than anybody else. I showed him the door and never went back.