Honoring “NO ROBOTS, NO FOLLOW” was purely voluntary, as you found out.
What a trove on FR! Could be used for good with all the rich content and could be used for evil.
There were so many search engines back when — all of them started as crawlers of the web, but one was actually called WebCrawler. AltaVista was the first really nice one that worked well, but Yahoo started as a giant manually edited web index, as did all the “so-and-so’s list” sites. Eventually Google came along and within a year or two AV was moribund and got bought by some other entity. The Wayback Machine / Web Archive was one site that ignored “NO ROBOTS, NO FOLLOW”, and over the years I’ve mostly been glad about that. :^)