2 Commits

Author	SHA1	Message	Date
Anubhav Patel	3a7b949d6d	Adds integration with Protego robots.txt parser (#3935)	2019-08-27 09:41:31 +02:00
Anubhav Patel	8e813953bd	[MRG+1] [GSoC 2019] Interface for robots.txt parsers (#3796) ... Make the robots.txt parser configurable through the new ROBOTSTXT_PARSER setting, support the Reppy and Robotexclusionrulesparser parsers, and allow implementing custom robots.txt parsers.	2019-08-02 09:43:29 +02:00