RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Update docs/topics/downloader-middleware.rst 

Co-Authored-By: elacuesta <elacuesta@users.noreply.github.com>
This commit is contained in:
Vostretsov Nikita 2019-09-09 17:04:13 +03:00 committed by GitHub
parent e418554c21
commit 38828d3fd4
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
docs/topics/downloader-middleware.rst
View File

@ -1021,7 +1021,7 @@ RobotFileParser
~~~~~~~~~~~~~~~
`RobotFileParser <https://docs.python.org/3.7/library/urllib.robotparser.html>`_ is
Python's inbuilt robots.txt_ parser. The parser is fully compliant with `Martijn Koster's
Python's built-in robots.txt_ parser. The parser is fully compliant with `Martijn Koster's
1996 draft specification <http://www.robotstxt.org/norobots-rfc.txt>`_. It lacks
support for wildcard matching.