From 171fa1cd106f5a4ba34e6d57bdd65c8688564b82 Mon Sep 17 00:00:00 2001
From: Vostretsov Nikita <whalebot.helmsman@gmail.com>
Date: Tue, 10 Sep 2019 09:59:36 +0000
Subject: [PATCH] documentation rework

---
 docs/topics/downloader-middleware.rst | 116 +++++++++++++++++---------
 1 file changed, 76 insertions(+), 40 deletions(-)

diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst
index 192bfd19a..52be8ded2 100644
--- a/docs/topics/downloader-middleware.rst
+++ b/docs/topics/downloader-middleware.rst
@@ -1000,10 +1000,10 @@ RobotsTxtMiddleware
 
     Scrapy ships with support for the following robots.txt_ parsers:
 
-    * :ref:`RobotFileParser <python-robotfileparser>` (default)
+    * :ref:`Protego <protego-parser>` (default)
+    * :ref:`RobotFileParser <python-robotfileparser>`
     * :ref:`Reppy <reppy-parser>`
     * :ref:`Robotexclusionrulesparser <rerp-parser>`
-    * :ref:`Protego <protego-parser>`
 
     You can change the robots.txt_ parser with the :setting:`ROBOTSTXT_PARSER`
     setting. Or you can also :ref:`implement support for a new parser <support-for-new-robots-parser>`.
@@ -1015,50 +1015,78 @@ If :attr:`Request.meta <scrapy.http.Request.meta>` has
 the request will be ignored by this middleware even if
 :setting:`ROBOTSTXT_OBEY` is enabled.
 
+Parsers varies in several aspects:
+
+* Language of implementation
+
+* Supported specification
+
+* Support for wildcard matching
+
+* usage of length based rule: in particular for ``Allow`` and
+  ``Disallow`` directives, where the most specific rule based on the length of
+  the path trumps the less specific (shorter) rule
+
+
+.. _protego-parser:
+
+Protego parser
+~~~~~~~~~~~~~~
+
+based on `Protego <https://github.com/scrapy/protego>`_:
+
+* implemented in Python
+
+* is compliant with `Google's Robots.txt Specification
+  <https://developers.google.com/search/reference/robots_txt>`_
+
+* supports wildcard matching
+
+* uses the length based rule,
+
+Scrapy uses this parser by default.
+
 .. _python-robotfileparser:
 
 RobotFileParser
 ~~~~~~~~~~~~~~~
 
-`RobotFileParser <https://docs.python.org/3.7/library/urllib.robotparser.html>`_ is
-Python's built-in robots.txt_ parser. The parser is fully compliant with `Martijn Koster's
-1996 draft specification <http://www.robotstxt.org/norobots-rfc.txt>`_. It lacks
-support for wildcard matching.
+based on `RobotFileParser
+<https://docs.python.org/3.7/library/urllib.robotparser.html>`_:
+
+* is Python's built-in robots.txt_ parser.
+
+* is compliant with `Martijn Koster's 1996 draft specification
+  <http://www.robotstxt.org/norobots-rfc.txt>`_.
+
+* lacks support for wildcard matching.
+
+* doesn't use the length based rule,
+
+It is faster than Protego and backward-compatible with versions of Scrapy before 1.8.0 .
 
 In order to use this parser, set:
 
 * :setting:`ROBOTSTXT_PARSER` to ``scrapy.robotstxt.PythonRobotParser``
 
-.. _rerp-parser:
-
-Robotexclusionrulesparser
-~~~~~~~~~~~~~~~~~~~~~~~~~
-
-`Robotexclusionrulesparser <http://nikitathespider.com/python/rerp/>`_ is fully compliant
-with `Martijn Koster's 1996 draft specification <http://www.robotstxt.org/norobots-rfc.txt>`_,
-with support for wildcard matching.
-
-In order to use this parser:
-
-* Install `Robotexclusionrulesparser <http://nikitathespider.com/python/rerp/>`_ by running
-  ``pip install robotexclusionrulesparser``
-
-* Set :setting:`ROBOTSTXT_PARSER` setting to
-  ``scrapy.robotstxt.RerpRobotParser``
-
 .. _reppy-parser:
 
 Reppy parser
 ~~~~~~~~~~~~
 
-`Reppy <https://github.com/seomoz/reppy/>`_ is a Python wrapper around `Robots Exclusion
-Protocol Parser for C++ <https://github.com/seomoz/rep-cpp>`_. The parser is fully compliant
-with `Martijn Koster's 1996 draft specification <http://www.robotstxt.org/norobots-rfc.txt>`_,
-with support for wildcard matching. Unlike
-`RobotFileParser <https://docs.python.org/3.7/library/urllib.robotparser.html>`_ and
-`Robotexclusionrulesparser <http://nikitathespider.com/python/rerp/>`_, it uses the length based
-rule, in particular for ``Allow`` and ``Disallow`` directives, where the most specific
-rule based on the length of the path trumps the less specific (shorter) rule.
+based on `Reppy <https://github.com/seomoz/reppy/>`_:
+
+* is a Python wrapper around `Robots Exclusion Protocol Parser for C++
+  <https://github.com/seomoz/rep-cpp>`_.
+
+* is compliant with `Martijn Koster's 1996 draft specification
+  <http://www.robotstxt.org/norobots-rfc.txt>`_.
+
+* supports wildcard matching
+
+* uses the length based rule,
+
+Native implementation provides better speed than Protego.
 
 In order to use this parser:
 
@@ -1067,21 +1095,29 @@ In order to use this parser:
 * Set :setting:`ROBOTSTXT_PARSER` setting to
   ``scrapy.robotstxt.ReppyRobotParser``
 
-.. _protego-parser:
+.. _rerp-parser:
 
-Protego parser
-~~~~~~~~~~~~~~
+Robotexclusionrulesparser
+~~~~~~~~~~~~~~~~~~~~~~~~~
 
-`Protego <https://github.com/scrapy/protego>`_ is a pure-Python robots.txt_ parser.
-The parser is fully compliant with `Google's Robots.txt Specification
-<https://developers.google.com/search/reference/robots_txt>`_ hence supports wildcard
-matching, and uses the length based rule similar to `Reppy <https://github.com/seomoz/reppy/>`_.
-Scrapy uses this parser by default.
+based on `Robotexclusionrulesparser <http://nikitathespider.com/python/rerp/>`_:
+
+* implemented in Python
+
+* is compliant with `Martijn Koster's 1996 draft specification
+  <http://www.robotstxt.org/norobots-rfc.txt>`_.
+
+* supports wildcard matching
+
+* doesn't use the length based rule,
 
 In order to use this parser:
 
+* Install `Robotexclusionrulesparser <http://nikitathespider.com/python/rerp/>`_ by running
+  ``pip install robotexclusionrulesparser``
+
 * Set :setting:`ROBOTSTXT_PARSER` setting to
-  ``scrapy.robotstxt.ProtegoRobotParser``
+  ``scrapy.robotstxt.RerpRobotParser``
 
 .. _support-for-new-robots-parser: