I would have been fine with traveling back to Silver Spring, MD on occasion, but they wanted me back there every week day I had a shift. I could not do that, so I will find another job.
With all remote work, if somebody asks me occasionally to go to the site, wherever it is, I am willing to go to the site, even if it is in Crookifornia, or, in some remote jobs, even in London or Berlin.
Yes they should have offered you once/twice a week in the office. Balance. The best is both worlds and the happiest and most productive employees I predict.