We pull the shoes on all of our horses when they aren't being ridden, with the exception of a mare who goes lame without front shoes.
I don't think you're wrong for pulling them, although I don't believe barefoot is necessarily better. It just depends on what your ground is like, how your horse is being used, where he's being ridden type stuff.
Has your horse been sick at all or out on grass? I've forgotten where you live.
How's yours? Is he still out the window or have you put him back in his stall?