A program that visits websites and reads their pages and content, by following hyperlinks to other pages in order to index the entries on search engines. Like a spider actually crawling thread by thread, or link by link through its web, a search engine spider does the same but through hyperlinks.
