diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index 0845ef6e4..539832618 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -513,7 +513,7 @@ defines the methods described below. .. method:: open_spider(spider) - This method gets called after a spider has been opened for crawling. It handles + This method gets called after a spider has been opened for crawling. It handles the :signal:`open_spider ` signal. :param spider: the spider which has been opened @@ -521,8 +521,8 @@ defines the methods described below. .. method:: close_spider(spider) - This method gets called after a spider has been closed. It handles - the :signal:`close_spider ` signal. + This method gets called after a spider has been closed. It handles + the :signal:`close_spider ` signal. :param spider: the spider which has been closed :type spider: :class:`~scrapy.spiders.Spider` object @@ -1000,10 +1000,10 @@ RobotsTxtMiddleware Scrapy ships with support for the following robots.txt_ parsers: - * :ref:`RobotFileParser ` (default) + * :ref:`Protego ` (default) + * :ref:`RobotFileParser ` * :ref:`Reppy ` * :ref:`Robotexclusionrulesparser ` - * :ref:`Protego ` You can change the robots.txt_ parser with the :setting:`ROBOTSTXT_PARSER` setting. Or you can also :ref:`implement support for a new parser `. @@ -1015,50 +1015,81 @@ If :attr:`Request.meta ` has the request will be ignored by this middleware even if :setting:`ROBOTSTXT_OBEY` is enabled. +Parsers vary in several aspects: + +* Language of implementation + +* Supported specification + +* Support for wildcard matching + +* Usage of `length based rule `_: + in particular for ``Allow`` and ``Disallow`` directives, where the most + specific rule based on the length of the path trumps the less specific + (shorter) rule + +Performance comparison of different parsers is available at `the following link +`_. + +.. _protego-parser: + +Protego parser +~~~~~~~~~~~~~~ + +Based on `Protego `_: + +* implemented in Python + +* is compliant with `Google's Robots.txt Specification + `_ + +* supports wildcard matching + +* uses the length based rule + +Scrapy uses this parser by default. + .. _python-robotfileparser: RobotFileParser ~~~~~~~~~~~~~~~ -`RobotFileParser `_ is -Python's inbuilt robots.txt_ parser. The parser is fully compliant with `Martijn Koster's -1996 draft specification `_. It lacks -support for wildcard matching. Scrapy uses this parser by default. +Based on `RobotFileParser +`_: + +* is Python's built-in robots.txt_ parser + +* is compliant with `Martijn Koster's 1996 draft specification + `_ + +* lacks support for wildcard matching + +* doesn't use the length based rule + +It is faster than Protego and backward-compatible with versions of Scrapy before 1.8.0. In order to use this parser, set: * :setting:`ROBOTSTXT_PARSER` to ``scrapy.robotstxt.PythonRobotParser`` -.. _rerp-parser: - -Robotexclusionrulesparser -~~~~~~~~~~~~~~~~~~~~~~~~~ - -`Robotexclusionrulesparser `_ is fully compliant -with `Martijn Koster's 1996 draft specification `_, -with support for wildcard matching. - -In order to use this parser: - -* Install `Robotexclusionrulesparser `_ by running - ``pip install robotexclusionrulesparser`` - -* Set :setting:`ROBOTSTXT_PARSER` setting to - ``scrapy.robotstxt.RerpRobotParser`` - .. _reppy-parser: Reppy parser ~~~~~~~~~~~~ -`Reppy `_ is a Python wrapper around `Robots Exclusion -Protocol Parser for C++ `_. The parser is fully compliant -with `Martijn Koster's 1996 draft specification `_, -with support for wildcard matching. Unlike -`RobotFileParser `_ and -`Robotexclusionrulesparser `_, it uses the length based -rule, in particular for ``Allow`` and ``Disallow`` directives, where the most specific -rule based on the length of the path trumps the less specific (shorter) rule. +Based on `Reppy `_: + +* is a Python wrapper around `Robots Exclusion Protocol Parser for C++ + `_ + +* is compliant with `Martijn Koster's 1996 draft specification + `_ + +* supports wildcard matching + +* uses the length based rule + +Native implementation, provides better speed than Protego. In order to use this parser: @@ -1067,22 +1098,29 @@ In order to use this parser: * Set :setting:`ROBOTSTXT_PARSER` setting to ``scrapy.robotstxt.ReppyRobotParser`` -.. _protego-parser: +.. _rerp-parser: -Protego parser -~~~~~~~~~~~~~~ +Robotexclusionrulesparser +~~~~~~~~~~~~~~~~~~~~~~~~~ -`Protego `_ is a pure-Python robots.txt_ parser. -The parser is fully compliant with `Google's Robots.txt Specification -`_ hence supports wildcard -matching, and uses the length based rule similar to `Reppy `_. +Based on `Robotexclusionrulesparser `_: + +* implemented in Python + +* is compliant with `Martijn Koster's 1996 draft specification + `_ + +* supports wildcard matching + +* doesn't use the length based rule In order to use this parser: -* Install `Protego `_ by running ``pip install protego`` +* Install `Robotexclusionrulesparser `_ by running + ``pip install robotexclusionrulesparser`` * Set :setting:`ROBOTSTXT_PARSER` setting to - ``scrapy.robotstxt.ProtegoRobotParser`` + ``scrapy.robotstxt.RerpRobotParser`` .. _support-for-new-robots-parser: diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 943ba13ee..75e0af63b 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -951,7 +951,7 @@ LOGSTATS_INTERVAL Default: ``60.0`` -The interval (in seconds) between each logging printout of the stats +The interval (in seconds) between each logging printout of the stats by :class:`~scrapy.extensions.logstats.LogStats`. .. setting:: MEMDEBUG_ENABLED @@ -1165,7 +1165,7 @@ If enabled, Scrapy will respect robots.txt policies. For more information see ROBOTSTXT_PARSER ---------------- -Default: ``'scrapy.robotstxt.PythonRobotParser'`` +Default: ``'scrapy.robotstxt.ProtegoRobotParser'`` The parser backend to use for parsing ``robots.txt`` files. For more information see :ref:`topics-dlmw-robots`. diff --git a/requirements-py2.txt b/requirements-py2.txt index 9e6944240..dde8d1c9c 100644 --- a/requirements-py2.txt +++ b/requirements-py2.txt @@ -1,6 +1,7 @@ parsel>=1.5.0 PyDispatcher>=2.0.5 w3lib>=1.17.0 +protego>=0.1.15 pyOpenSSL>=16.2.0 # Earlier versions fail with "AttributeError: module 'lib' has no attribute 'SSL_ST_INIT'" queuelib>=1.4.2 # Earlier versions fail with "AttributeError: '...QueueTest' object has no attribute 'qpath'" diff --git a/requirements-py3.txt b/requirements-py3.txt index cd183a525..2c98e6f6d 100644 --- a/requirements-py3.txt +++ b/requirements-py3.txt @@ -2,6 +2,7 @@ parsel>=1.5.0 PyDispatcher>=2.0.5 Twisted>=17.9.0 w3lib>=1.17.0 +protego>=0.1.15 pyOpenSSL>=16.2.0 # Earlier versions fail with "AttributeError: module 'lib' has no attribute 'SSL_ST_INIT'" queuelib>=1.4.2 # Earlier versions fail with "AttributeError: '...QueueTest' object has no attribute 'qpath'" diff --git a/scrapy/settings/default_settings.py b/scrapy/settings/default_settings.py index 05ab4b628..9c22999cb 100644 --- a/scrapy/settings/default_settings.py +++ b/scrapy/settings/default_settings.py @@ -246,7 +246,7 @@ RETRY_HTTP_CODES = [500, 502, 503, 504, 522, 524, 408, 429] RETRY_PRIORITY_ADJUST = -1 ROBOTSTXT_OBEY = False -ROBOTSTXT_PARSER = 'scrapy.robotstxt.PythonRobotParser' +ROBOTSTXT_PARSER = 'scrapy.robotstxt.ProtegoRobotParser' ROBOTSTXT_USER_AGENT = None SCHEDULER = 'scrapy.core.scheduler.Scheduler' diff --git a/setup.py b/setup.py index 37892cfbf..850456503 100644 --- a/setup.py +++ b/setup.py @@ -77,6 +77,7 @@ setup( 'six>=1.10.0', 'w3lib>=1.17.0', 'zope.interface>=4.1.3', + 'protego>=0.1.15', ], extras_require=extras_require, ) diff --git a/tox.ini b/tox.ini index cc845faf1..ffe7360d3 100644 --- a/tox.ini +++ b/tox.ini @@ -33,6 +33,7 @@ deps = cssselect==0.9.1 lxml==3.5.0 parsel==1.5.0 + Protego==0.1.15 PyDispatcher==2.0.5 pyOpenSSL==16.2.0 queuelib==1.4.2 @@ -69,6 +70,7 @@ deps = cssselect==0.9.1 lxml==3.5.0 parsel==1.5.0 + Protego==0.1.15 PyDispatcher==2.0.5 pyOpenSSL==16.2.0 queuelib==1.4.2 @@ -125,7 +127,6 @@ deps = {[testenv:py35]deps} reppy robotexclusionrulesparser - protego [testenv:py27-extra-deps] basepython = python2.7 @@ -133,4 +134,3 @@ deps = {[testenv]deps} reppy robotexclusionrulesparser - protego \ No newline at end of file