I can relate to that, at one time I suppose I was agnostic. (when I was young)
Now, I am more of a Deist, or a church of one, so to speak.
There has always been something about the Sunday church routine that bothered me. It is more the fact that people get religion on Sunday then go home and sin some more.
It just bothers me.
Also the fact that the powers in the church are constantly doing and supporting liberal causes that I can not support or contribute to.
Considering this and that I am not a very social person, the organized church makes me uncomfortable, and I truly tried many of them years ago.
My response when asked to go to church was always, "do I have to?"
It seems I have never gotten over that.(This is probably more than you want to know, but what the heck)