RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Document update URLLENGTH_LIMIT

This commit is contained in:
Deepanshu 2021-10-01 01:47:05 +05:30
parent 7d557f898c
commit 74f146bbe0
2 changed files with 7 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
docs/topics/settings.rst
View File

@ -1645,8 +1645,12 @@ Default: ``2083``
Scope: ``spidermiddlewares.urllength``
The maximum URL length to allow for crawled URLs. For more information about
the default value for this setting see: https://support.microsoft.com/en-us/topic/maximum-url-length-is-2-083-characters-in-internet-explorer-174e7c8a-6666-f4e0-6fd6-908b53c12246
The maximum URL length to allow for crawled URLs. You can set this to ``0``
to disable :class:`~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware` for working
with URLs longer than the default value. The default limit acts as a stopping condition in case of
URLs of increasing length, usually caused by a loop.
For more information about the default value
for this setting see: https://support.microsoft.com/en-us/topic/maximum-url-length-is-2-083-characters-in-internet-explorer-174e7c8a-6666-f4e0-6fd6-908b53c12246
.. setting:: USER_AGENT

1
docs/topics/spider-middleware.rst
View File

@ -440,4 +440,5 @@ UrlLengthMiddleware
settings (see the settings documentation for more info):
* :setting:`URLLENGTH_LIMIT` - The maximum URL length to allow for crawled URLs.
If ``0``, then :class:`~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware` is disabled.