RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

remove periods

This commit is contained in:
Vostretsov Nikita 2019-09-17 06:28:33 +00:00
parent 50c85e3114
commit b792dba528
1 changed files with 6 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
docs/topics/downloader-middleware.rst
View File

@ -1054,12 +1054,12 @@ RobotFileParser
Based on `RobotFileParser
<https://docs.python.org/3.7/library/urllib.robotparser.html>`_:
* is Python's built-in robots.txt_ parser.
* is Python's built-in robots.txt_ parser
* is compliant with `Martijn Koster's 1996 draft specification
<http://www.robotstxt.org/norobots-rfc.txt>`_.
<http://www.robotstxt.org/norobots-rfc.txt>`_
* lacks support for wildcard matching.
* lacks support for wildcard matching
* doesn't use the length based rule
@ -1077,10 +1077,10 @@ Reppy parser
Based on `Reppy <https://github.com/seomoz/reppy/>`_:
* is a Python wrapper around `Robots Exclusion Protocol Parser for C++
<https://github.com/seomoz/rep-cpp>`_.
<https://github.com/seomoz/rep-cpp>`_
* is compliant with `Martijn Koster's 1996 draft specification
<http://www.robotstxt.org/norobots-rfc.txt>`_.
<http://www.robotstxt.org/norobots-rfc.txt>`_
* supports wildcard matching
@ -1105,7 +1105,7 @@ Based on `Robotexclusionrulesparser <http://nikitathespider.com/python/rerp/>`_:
* implemented in Python
* is compliant with `Martijn Koster's 1996 draft specification
<http://www.robotstxt.org/norobots-rfc.txt>`_.
<http://www.robotstxt.org/norobots-rfc.txt>`_
* supports wildcard matching