1 Commits

Author	SHA1	Message	Date
Anubhav Patel	8e813953bd	[MRG+1] [GSoC 2019] Interface for robots.txt parsers (#3796) ... Make the robots.txt parser configurable through the new ROBOTSTXT_PARSER setting, support the Reppy and Robotexclusionrulesparser parsers, and allow implementing custom robots.txt parsers.	2019-08-02 09:43:29 +02:00