Not necessarily. A surprisingly large number of people are hypersensitive to certain fabrics and textures. The medical term for it is "tactile defensiveness", which seems to be a misnomer to me; "tactile hypersensitivity" would be a more accurate term for it.
It is often associated with ADHD, Asperger's etc, and some autistic kids have it *really bad*. However, ADHD and mild-Asperger types are far more numerous in the population, and when we say that certain clothes, certain fabrics, etc, just don't feel right on our skin, we are NOT KIDDING. For those of us who have this condition, clothes that are made of certain fabrics are just torment to wear, even if they "fit" perfectly.
As it happens, by perverse fate, in our culture most of the "formal" clothes just so happen to be the most uncomfortable, so we are often accused of "dressing down" for some ideological motive, or due to just plain laziness, when we are actually driven to make our clothing choices by purely physical, neurological considerations instead. If things were reversed -- if jeans, soft flannels, etc, were considered "formal", we'd be accused of being "hoity-toity" for dressing "UP" all the time!
As it happens, by perverse fate, in our culture most of the "formal" clothes just so happen to be the most uncomfortable