Oh yeah, good call. You can also use WebCrawer, which goes way back. http://www.webcrawler.com/
Webcrawler! I haven’t thought about that one in years!
Another oldie, http://metasearch.com/