“I’d rather pay a bit more for higher quality US goods and have fewer shirts or trousers ...”
You can still buy a made-in-the-USA all-cotton dress shirt, but it will most likely cost you +$100. I like Gitman Bro. shirts if/when I find them at a year-end or clearance discount. Some online merchants sell vintage Sero shirts at around $50. Here is one list of US-made shirts:
http://www.americanologist.com/2014/12/15-sources-of-quality-mens-dress-shirts-made-in-usa/
I’m willing to pay more for things that last longer. And the truth is, I’m content with 3 or 4 sets of clothing that mix or match.
A good friend ran a textile plant down south that went overseas, and they were by no means overpaying their employees.