My wife hates it when I make this suggestion, but that sure sounds like it's the clothes more than the man that you find attractive!
It's what the expensive suit implies about the man's character, taste, and high social status that women find attractive.
It's not that simple. It's the image, the romance of what the clothes say about him we're attracted to, perhaps... In clothes, he's an image that evokes some emotional response. In clothes, men are cowboys, or construction workers, or James Bond, or soldiers, or our guy in jeans, or dressed to go someplace nice, if suits are unusual for him.... walking around nekkid, guys are simian and uncivilized, and for all we know, might be about to scratch themselves ;~D
You should have learned by now that women are much more complicated than a simple visual reaction. Women don't want to see male nakedness unless she's about to have sex with him... and even then, it's better if he doesn't just walk in like that. :~D
I think it's both. The clothes do make the man.