The Houyhnhnms recognize that Gulliver is a Yahoo, a human, but for a time think he's more civilized than most. In discussing civilized (i.e. European) Houyhnhnms with Master Horse he pillories most of the major character defects of modern mankind (e.g. warfare, greed, dishonesty, avarice, etc.), only convincing him in the end that the Houyhnhnm Land Yahoos are superior to Europeans because they are honest about their brutishness.
Yahoos also expose what humans may look like in the state of nature, a very popular topic/theory in those days, and indeed in the prestent world. In the state of nature, Man is always depicted as a perfect innocent, in a society that does not know war, slavery, in many accounts "patriarchy" or any other foible the author dislikes. The Yahoo character pillories the various state of nature theories in taking the position that in the state of nature Man really is a brute.
In other words, the Yahoos are a foil to expose the defects of civilized European society that European manners conceal. Nothing whatsoever to do with identifying Sasquatch or any real creature.