There were so many search engines back when — all of them started as crawlers of the web, but one was actually called WebCrawler. AltaVista was the first really nice one that worked well, but Yahoo started as a giant manually edited web index, as did all the “so-and-so’s list” sites. Eventually Google came along and within a year or two AV was moribund and got bought by some other entity. The Wayback Machine / Web Archive was one site that ignored “NO ROBOTS, NO FOLLOW”, and over the years I’ve mostly been glad about that. :^)
AltaVista was located below my company on a hill in Palo Alto. I used AV extensively and switched to Google not long after i learned about it.