Spider (computer), computer program that automatically monitors documents (called Web pages) on the World Wide Web (WWW); also called robot, softbot, wanderer, crawler, and fish.
Most Web pages include at least one link (an automatic connection) to another Web page, and some include hundreds of links. A spider takes advantage of this structure by starting at one Web page and working its way out by following every link on a Web page and then following every link provided by the new Web pages.
Some spiders save the URL (Uniform Resource Locator), or address, of every Web page they visit. These spiders are used by search engines to build indexes of Web pages that users can access to search for information on a particular topic. Indexing spiders, as they are called, often also store the title and partial or complete text of a Web page so users can do more detailed searches.
Some spiders store only URLs of Web pages that have not been listed yet in order to update lists or provide lists of new Web pages. Some spiders make note of URLs that are no longer valid in order to correct lists.
Friday, January 16, 2009
Subscribe to:
Post Comments (Atom)
related story:
Search Engine
Service Provider
Tel-Net
Spider Web
Microsoft
Mark-up Language
No comments:
Post a Comment