5 Commits

Author	SHA1	Message	Date
Adrián Chaves	a2bf340bab	Remove unused imports	2019-11-21 14:21:17 +01:00
Adrián Chaves	f261cf65e9	Add missing blank lines between functions and classes ... Also fixed 2 unrelated Flake8 issues	2019-11-18 17:16:09 +01:00
Andrey Rakhmatullin	a138fb05d4	Replace to_native_str calls with to_unicode.	2019-11-14 13:45:39 +05:00
Anubhav Patel	3a7b949d6d	Adds integration with Protego robots.txt parser (#3935)	2019-08-27 09:41:31 +02:00
Anubhav Patel	8e813953bd	[MRG+1] [GSoC 2019] Interface for robots.txt parsers (#3796) ... Make the robots.txt parser configurable through the new ROBOTSTXT_PARSER setting, support the Reppy and Robotexclusionrulesparser parsers, and allow implementing custom robots.txt parsers.	2019-08-02 09:43:29 +02:00