Anubhav Patel 8e813953bd [MRG+1] [GSoC 2019] Interface for robots.txt parsers (#3796) ... Make the robots.txt parser configurable through the new ROBOTSTXT_PARSER setting, support the Reppy and Robotexclusionrulesparser parsers, and allow implementing custom robots.txt parsers.		2019-08-02 09:43:29 +02:00
..
__init__.py	Suggested changes	2019-07-15 12:37:09 -03:00
default_settings.py	[MRG+1] [GSoC 2019] Interface for robots.txt parsers (#3796)	2019-08-02 09:43:29 +02:00
deprecated.py	Include deprecated warning	2016-07-25 18:05:51 +05:30