Thread Options

The threads page allows you to set the number of download threads that WebReaper will use, and to configure how WebReaper behaves when the download is completed.

Maximum Number of threads

Webreaper uses several threads to download links. This makes it faster, as one thread can continue work whilst the another waits for a response from the internet server from which the file is being downloaded. Depending on your machine, network connection and the site you are downloading from, different numbers of threads can give different performance.

For example, if lots of small files are being downloaded, having many threads running at a time can be faster - many requests will be sent out to the internet, and so it's useful have more work done whilst waiting for responses. If the site from which you are downloading has a few large files, fewer threads may work faster - once the initial request has been made the file can be downloaded using the entire available bandwidth.

Note that IE (and therefore WebReaper) can only have two connections to an HTTP server open at any one time. This is due to the HTTP standards and is implemented to stop browsers/robots overloading the server. So you will find that even if you have more than two threads running, only two may be active at any one time. It is possible (though not recommended) to override this - more details can be found on Microsoft's MSDN site.

Inter-object Delay

In order not to overload servers whilst reaping a site from them, it is courteous to pause in between pages/objects. This stops webreaper from hogging the entire host server's network bandwidth. To do this, set WebReaper to use just one thread, and then specify an inter-object dely of one or two seconds. This field is disabled if more than one thread is to be used.

Maximum Download Time

Allows you to specify a maximum total time for the download to take. The time remaining will be displayed on the status bar during the download.

When Download Completes

Allow Current Files to Finish

If a download is aborted before it completes (by pressing the Stop button on the toolbar), WebReaper will abort any files which are currently downloading and stop as quickly as possible. However, this may result in partially-downloaded files being left locally. By checking this option, you can ensure that WebReaper will complete all files currently being downloaded before the session is aborted.