Connections

View thumbnailView full size image

Concurrent requests

This controls the number of retrieval threads that Web Spider uses. In other words, how many page requests will be processed simultaneously. The default of 8 is suitable for most situations. Depending on the speed of your connection, more could increase the speed of spidering. For troubleshooting, you may want to limit this to a single connection thread until you are satisfied that Perceptive Search WebSite Spider is working correctly. Although very unusual, some web servers may block large numbers of connections from the same IP address, assuming it a Denial of Service attack.

Retrieval timeout

If a requested page results in no reply at all, Perceptive Search WebSite Spider will wait for the specified time (the 'timeout') until retrying (if set to do so above). If it fails after the final try (which may be the first if max retries = 0), it will log the failure.

Number of retries

If a requested page results in a 'Page not found' or other error, Perceptive Search WebSite Spider can retry the request. The number set here is the maximum number of retries before giving up and logging a failure. (Default is 1.)

Pause between requests

You can also nominate that you want the Web Spider to pause between pages for a specified time. This may be desirable if you do not want the Web Spider to dominate your machine or network resources (or the web server is particularly slow), and are happy for traversal to take longer for the benefit of better resource sharing.

User Agent

When a browser requests a page from a web server, it supplies certain information to it in the form of 'headers', including the type and version of browser making the request. Some web servers will tailor their responses based on these headers, by supplying a page specifically for browsers with the capability to display certain elements, or with a certain level of scripting. However, this is more often used for logging information such as the browser agent, operating system and geographical location of requests.

There are several options available:

• Perceptive Search Webspider
  The user agent will identify itself as Perceptive Search spider.
• Internet Explorer
  This will supply generic Internet Explorer browser headers to the web server.
• Netscape Navigator
  This will supply generic Netscape Navigator browser headers to the web server.
• None
  This supplies a blank agent name.
• Other...
  This allows you to specify whatever you like as the browser agent header.

Connect via proxy server

If you need to connect via a proxy server from your machine to the remote site, enter its IP address or URL in the Proxy Address input box, and select the Proxy Enabled checkbox. You will also need to provide the port number for the proxy. If you have Proxy information already set in your Microsoft Internet Explorer configuration, then you do not need set Proxy information within Perceptive Search WebSite Spider. Perceptive Search WebSite Spider will follow the configuration for Internet Explorer. However, if you are using Netscape Navigator and have configured Proxy information, then you will also need to configure Proxy details within Perceptive Search WebSite Spider.

Proxy account

If your proxy server requires authentication, type your username and password here.