Anubhav Patel 8e813953bd [MRG+1] [GSoC 2019] Interface for robots.txt parsers (#3796) ... Make the robots.txt parser configurable through the new ROBOTSTXT_PARSER setting, support the Reppy and Robotexclusionrulesparser parsers, and allow implementing custom robots.txt parsers.		2019-08-02 09:43:29 +02:00
..
__init__.py	Move scrapy/contrib/downloadermiddleware to scrapy/downloadermiddlewares	2015-04-29 21:24:30 -03:00
ajaxcrawl.py	Remove duplicate code now handled by newer w3lib	2016-03-30 14:58:03 +00:00
chunked.py	Use https links wherever possible	2017-10-28 16:24:40 +05:30
cookies.py	PY3 port COOKIES_DEBUG and add tests	2015-08-03 16:28:29 -03:00
decompression.py	Move scrapy/contrib/downloadermiddleware to scrapy/downloadermiddlewares	2015-04-29 21:24:30 -03:00
defaultheaders.py	Move scrapy.utils.conf.remove_none_values to s.u.python.without_none_values	2015-10-27 12:43:33 +01:00
downloadtimeout.py	Move scrapy/contrib/downloadermiddleware to scrapy/downloadermiddlewares	2015-04-29 21:24:30 -03:00
httpauth.py	PY3 fix HttpAuthMiddleware tests	2015-09-01 04:00:26 +05:00
httpcache.py	Use https links wherever possible	2017-10-28 16:24:40 +05:30
httpcompression.py	Do not silently fail on gzip unzipping	2017-03-07 11:02:46 +01:00
httpproxy.py	the strip() isnt needed	2018-12-04 10:57:51 +01:00
redirect.py	Implement the METAREFRESH_IGNORE_TAGS setting	2019-05-08 12:52:29 +02:00
retry.py	Remove the unexisting retry_complete signal from the documentation	2019-03-08 14:21:00 +01:00
robotstxt.py	[MRG+1] [GSoC 2019] Interface for robots.txt parsers (#3796)	2019-08-02 09:43:29 +02:00
stats.py	retry stats	2017-02-14 20:27:49 +05:00
useragent.py	PY3 fix UserAgentMiddleware tests	2015-09-01 04:11:33 +05:00