13 Commits

Author	SHA1	Message	Date
Andrey Rakhmatullin	5759b3f0f2	Drop Reppy.	2024-10-17 23:41:23 +05:00
Andrey Rakhmatullin	feb0b8f7dc	Add pyupgrade.	2024-06-05 22:57:18 +05:00
Lorenzo Verardo	7b37dcd80d	Handle robots.txt files not UTF-8 encoded	2024-04-04 12:22:50 +02:00
Marc Hernández	5360ba34bc	IOError and other cleanup (#4716)	2023-06-21 20:08:53 +02:00
Emmanuel Rondan	e211ec0aa2	adding black formatter to all the code	2022-11-29 11:30:46 -03:00
Marc Hernández	336f19f5cc	Change super syntax (#4707)	2020-08-04 20:42:01 +02:00
Adrián Chaves	890b2138a6	Remove the u prefix from strings	2020-07-30 13:39:30 +02:00
Marc Hernandez Cabot	a5de2c64e6	fix W291, W292, W293 (whitespaces)	2019-12-18 16:33:46 +01:00
Adrián Chaves	a2bf340bab	Remove unused imports	2019-11-21 14:21:17 +01:00
Adrián Chaves	f261cf65e9	Add missing blank lines between functions and classes ... Also fixed 2 unrelated Flake8 issues	2019-11-18 17:16:09 +01:00
Andrey Rakhmatullin	a138fb05d4	Replace to_native_str calls with to_unicode.	2019-11-14 13:45:39 +05:00
Anubhav Patel	3a7b949d6d	Adds integration with Protego robots.txt parser (#3935)	2019-08-27 09:41:31 +02:00
Anubhav Patel	8e813953bd	[MRG+1] [GSoC 2019] Interface for robots.txt parsers (#3796) ... Make the robots.txt parser configurable through the new ROBOTSTXT_PARSER setting, support the Reppy and Robotexclusionrulesparser parsers, and allow implementing custom robots.txt parsers.	2019-08-02 09:43:29 +02:00