RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

add link to google description of lenght-based rule

This commit is contained in:
Vostretsov Nikita 2019-09-17 06:34:33 +00:00
parent b792dba528
commit d39ef77e6b
1 changed files with 4 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
docs/topics/downloader-middleware.rst
View File

@ -1023,10 +1023,10 @@ Parsers vary in several aspects:
* Support for wildcard matching
* Usage of length based rule: in particular for ``Allow`` and
``Disallow`` directives, where the most specific rule based on the length of
the path trumps the less specific (shorter) rule
* Usage of `length based rule <https://developers.google.com/search/reference/robots_txt#order-of-precedence-for-group-member-lines>`_:
in particular for ``Allow`` and ``Disallow`` directives, where the most
specific rule based on the length of the path trumps the less specific
(shorter) rule
.. _protego-parser: