Well, he does have them “Yankee-blue” clothes.
Still is.
It's the "revenooer" angle that is relevant here. Outsiders are distrusted ~ as are neighbors you don't like.