You are exactly right. I was active in the Baptist Church until I was in my 40s and most of that time I was a pianist and teacher in the church. I became a Catholic, even a lector in the Catholic Church, in my 50-60s. Now, I belong only to the church in the body of Jesus Christ. Physical churches have become too liberal and I won't be in those physical churches just to show the public I go to church. Physical churches have removed themselves too far from Jesus Christ so I removed myself from them.
The church can be anywhere.........it is not brick and mortar, which many people think it is.
A really long time ago, a pastor said in the days of Jesus, if people where asked where the church was, the answer would be, “well, Jane is over there, Peter is over there.....”, today people would say “123 Main street”.
It is amazing to me how many people go to “church” to be served, rather than to serve.
So, I completely understand.