You nailed it.
As a sailor in the U.S. Navy, I've traveled the world. You can always spot the American tourists in foreign ports. They're the ones dressed like they're about to go mow the lawn. My wife is from Ukraine and the first time she went to a grocery store in America and saw other women wearing sweat pants, she couldn't believe it and called her friends back home!
And, yes, I'm occasionally guilty as well (though I've made it a habit to wear more button-down shirts and I try not to look "American" when we pull into foreign ports).