RDig Ferret based crawler and content extractor for building a full text index of a website's contents
Download

RDig Ranking & Summary

RDig Tags

index extract crawl http crawler content extractor web indexer

RDig Description

Ferret based crawler and content extractor for building a full text index of a website's contents RDig provides a content extraction and an HTTP crawler utilities to help building a site search for web sites or intranets. Internally, Ferret is used for the full text indexing. After creating a config file for your site, the index can be built with a single call to RDig.NOTE: RDig is developed and licensed under the terms of the MIT/X Consortium License. Requirements: · Ferret 0.1 or later · Hpricot 0.4 or later What's New in This Release: · Add max_depth option to crawler configuration for limiting the crawl to a specific depth · Add support for http proxies including basic authentication · Remove rubyful_soup support

RDig Related Software