diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index 0845ef6e4..be96425cf 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -513,7 +513,7 @@ defines the methods described below. .. method:: open_spider(spider) - This method gets called after a spider has been opened for crawling. It handles + This method gets called after a spider has been opened for crawling. It handles the :signal:`open_spider ` signal. :param spider: the spider which has been opened @@ -521,8 +521,8 @@ defines the methods described below. .. method:: close_spider(spider) - This method gets called after a spider has been closed. It handles - the :signal:`close_spider ` signal. + This method gets called after a spider has been closed. It handles + the :signal:`close_spider ` signal. :param spider: the spider which has been closed :type spider: :class:`~scrapy.spiders.Spider` object @@ -1020,10 +1020,10 @@ the request will be ignored by this middleware even if RobotFileParser ~~~~~~~~~~~~~~~ -`RobotFileParser `_ is -Python's inbuilt robots.txt_ parser. The parser is fully compliant with `Martijn Koster's +`RobotFileParser `_ is +Python's inbuilt robots.txt_ parser. The parser is fully compliant with `Martijn Koster's 1996 draft specification `_. It lacks -support for wildcard matching. Scrapy uses this parser by default. +support for wildcard matching. In order to use this parser, set: @@ -1074,13 +1074,12 @@ Protego parser `Protego `_ is a pure-Python robots.txt_ parser. The parser is fully compliant with `Google's Robots.txt Specification -`_ hence supports wildcard +`_ hence supports wildcard matching, and uses the length based rule similar to `Reppy `_. +Scrapy uses this parser by default. In order to use this parser: -* Install `Protego `_ by running ``pip install protego`` - * Set :setting:`ROBOTSTXT_PARSER` setting to ``scrapy.robotstxt.ProtegoRobotParser`` diff --git a/requirements-py2.txt b/requirements-py2.txt index 9e6944240..61176bdba 100644 --- a/requirements-py2.txt +++ b/requirements-py2.txt @@ -15,3 +15,4 @@ service_identity>=16.0.0 six>=1.10.0 Twisted>=16.0.0 zope.interface>=4.1.3 +protego diff --git a/requirements-py3.txt b/requirements-py3.txt index cd183a525..61e2e32d8 100644 --- a/requirements-py3.txt +++ b/requirements-py3.txt @@ -15,3 +15,4 @@ lxml>=3.5.0 service_identity>=16.0.0 six>=1.10.0 zope.interface>=4.1.3 +protego diff --git a/scrapy/settings/default_settings.py b/scrapy/settings/default_settings.py index 05ab4b628..9c22999cb 100644 --- a/scrapy/settings/default_settings.py +++ b/scrapy/settings/default_settings.py @@ -246,7 +246,7 @@ RETRY_HTTP_CODES = [500, 502, 503, 504, 522, 524, 408, 429] RETRY_PRIORITY_ADJUST = -1 ROBOTSTXT_OBEY = False -ROBOTSTXT_PARSER = 'scrapy.robotstxt.PythonRobotParser' +ROBOTSTXT_PARSER = 'scrapy.robotstxt.ProtegoRobotParser' ROBOTSTXT_USER_AGENT = None SCHEDULER = 'scrapy.core.scheduler.Scheduler' diff --git a/tox.ini b/tox.ini index cc845faf1..c3502c2ca 100644 --- a/tox.ini +++ b/tox.ini @@ -125,7 +125,6 @@ deps = {[testenv:py35]deps} reppy robotexclusionrulesparser - protego [testenv:py27-extra-deps] basepython = python2.7 @@ -133,4 +132,3 @@ deps = {[testenv]deps} reppy robotexclusionrulesparser - protego \ No newline at end of file