NWebCrawler This is a web crawler program written in C#.
Download

NWebCrawler Ranking & Summary

NWebCrawler Tags

grab grabber Java crawler Crawler web crawler data crawler Image Crawler LAN crawler share crawler file crawler website crawler link crawler internet crawler GDS crawler written email crawler Crawler framework crawler bar SiteMap Crawler Web Grabber thread count profile crawler crawler profile E-mail crawler email web crawler

NWebCrawler Description

The NWebCrawler application was developed to be a web crawler program written in C#. Main features: Configuable: thread count, waiting time, connection timeout, allow MIME types and priorities, download folders. Statstics information: URL count, total downloaded files, total downloaded bytes, CPU utility and available memory. Preferential crawler: user can set priority for MIME types (high, above, normal, below, low). Robust: 10+ URL normalization rules, crawler trap avoiding rules.

NWebCrawler Related Software