Robotexclusionrulesparser Free Download

Robotexclusionrulesparser Ranking & Summary

Ranking: Click at the star to rank
User Review: 0 ( times ) Download now
File Size: -
OS: -
License: Perl Artistic License
Price: FREE
Publisher: Philip Semanchuk

Robotexclusionrulesparser Tags

Robotexclusionrulesparser Description

An alternative to Python's robotparser module
Robotexclusionrulesparser is an alternative to the Python standard library module robotparser. It fetches and parses robots.txt files and can answer questions as to whether or not a given user agent is permitted to visit a certain URL., respect for Expires headers and understanding of Crawl-delay and Sitemap directives and wildcard syntax in path names.Complete documentation (including a comparison with the standard library module robotparser) is available in ReadMe.html. Requirements:
· Python

Robotexclusionrulesparser Related Software

Robotexclusionrulesparser Company

Philip Semanchuk

Company Information:

Robotexclusionrulesparser Reviews