Crawl Small and efficient HTTP crawler
Download

Crawl Ranking & Summary

Crawl Tags

parse crawler crawl http crawler web crawler crawl http

Crawl Description

Small and efficient HTTP crawler The crawl utility starts a depth-first traversal of the web at the specified URLs. It stores all JPEG images that match the configured constraints. Crawl is fairly fast and allows for graceful termination.After terminating crawl, it is possible to restart it at exactly the same spot where it was terminated. Crawl keeps a persistent database that allows multiple crawls without revisiting sites.Crawl contains only a few thousand lines of code, which makes it fairly easy to customize and debug.NOTE: Crawl is licensed and distributed under the terms of the BSD License. Here are some key features of "Crawl": · Saves encountered images or other media types · Media selection based on regular expressions and size contraints · Resume previous crawl after graceful termination · Persistent database of visited URLs · Very small and efficient code · Asynchronous DNS lookups · Supports robots.txt Requirements: · libevent

Crawl Related Software