Spidr Fast and easy to use web spider developed in Ruby
Download

Spidr Ranking & Summary

Spidr Tags

web spider spider spider website blacklist url blacklist whitelist

Spidr Description

Fast and easy to use web spider developed in Ruby Spidr is a versatile Ruby web spidering library that can spider a website, multiple domains, certain links or infinitely. Spidr is designed to be easy to use and fast. Here are some key features of "Spidr": · Custom User-Agent strings. · Custom proxy settings. · Black-list or white-list URLs based upon: Hostname, Port number, Full link, File extension · Provides call-backs for: Every visited URL, Every visited Page, Every visited URL that matches a specified pattern. Requirements: · Hpricot

Spidr Related Software