Indexing Websites

Many users want to index websites from the point of view of their end-user (a web surfer). The WebSite Spider module enables website indexing (and therefore searching) by crawling through the pages of a website, following the links and indexing the content of each page. Any information accessible via an HTTP Web server can be indexed and searched transparently.

What can WebSite Spider be used for?

• Indexing any web server, whether Internet, Intranet, or Extranet.
• Indexing data for which disk volume access is not available. For example, a Unix machine that is not accessible from the Local Area Network (LAN).
• Indexing an associated company's site to which you would like to provide umbrella search ability, or even a competitor's web site.
• Indexing Lotus Domino sites. The Web Spider will index the Lotus data in exactly the same way as they are presented to the web end-user.

Unlike most other web spidering search engines on the market, with WebSite Spider, documents found are presented with hits highlighted and full navigation between hits.

When used with the Perceptive Enterprise Search server, text is routed via the server for value-added automatic markup, while embedded graphics are routed directly to the original web site for optimum bandwidth utilization.

For more detailed information about Web Spider's operation see:

• Creating a new Website index
• Introduction to Website indexing