The point is that He is to be approached reverently, not casually. "Give of your best to the master" as the hymn goes. If jeans and a t-shirt are the only thing you have, if they are your "best", then I wouldn't have a problem.
The problem is when people who know better, and can do better, choose to be lazy about serving the Lord. When you put your personal comfort first, and serving Him second, that's selfish idolatry.
Knights wore ceremonail armor because it was their best. Togas and robes were worn, but I would venture to say that the ones worn were the best, as were the buckskins, as are today's Dockers.
However, when one approaches the Lord with a casual heart and in a flippant manner, it shows.
I see your point. When I go to church, I try to wear nice, clean clothes, but I think people get too worked up about it.