RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #5250 from iDeepverma/URLLENGTH_LIMIT-doc 

Documentation update for URLLENGTH_LIMIT
This commit is contained in:
Andrey Rahmatullin 2021-10-12 16:48:42 +05:00 committed by GitHub
parent 8a743ae43a f108800222
commit d90b2a2bfc
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 13 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

15
docs/topics/settings.rst
View File

@ -1645,8 +1645,19 @@ Default: ``2083``
Scope: ``spidermiddlewares.urllength`` Scope: ``spidermiddlewares.urllength``
The maximum URL length to allow for crawled URLs. For more information about The maximum URL length to allow for crawled URLs.
the default value for this setting see: https://support.microsoft.com/en-us/topic/maximum-url-length-is-2-083-characters-in-internet-explorer-174e7c8a-6666-f4e0-6fd6-908b53c12246
This setting can act as a stopping condition in case of URLs of ever-increasing
length, which may be caused for example by a programming error either in the
target server or in your code. See also :setting:`REDIRECT_MAX_TIMES` and
:setting:`DEPTH_LIMIT`.
Use ``0`` to allow URLs of any length.
The default value is copied from the `Microsoft Internet Explorer maximum URL
length`_, even though this setting exists for different reasons.
.. _Microsoft Internet Explorer maximum URL length: https://support.microsoft.com/en-us/topic/maximum-url-length-is-2-083-characters-in-internet-explorer-174e7c8a-6666-f4e0-6fd6-908b53c12246
.. setting:: USER_AGENT .. setting:: USER_AGENT

1
docs/topics/spider-middleware.rst
View File

@ -440,4 +440,3 @@ UrlLengthMiddleware
settings (see the settings documentation for more info): settings (see the settings documentation for more info):
* :setting:`URLLENGTH_LIMIT` - The maximum URL length to allow for crawled URLs. * :setting:`URLLENGTH_LIMIT` - The maximum URL length to allow for crawled URLs.