diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 2ab2020fa..4d3ae20cc 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -1645,8 +1645,12 @@ Default: ``2083`` Scope: ``spidermiddlewares.urllength`` -The maximum URL length to allow for crawled URLs. For more information about -the default value for this setting see: https://support.microsoft.com/en-us/topic/maximum-url-length-is-2-083-characters-in-internet-explorer-174e7c8a-6666-f4e0-6fd6-908b53c12246 +The maximum URL length to allow for crawled URLs. You can set this to ``0`` +to disable :class:`~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware` for working +with URLs longer than the default value. The default limit acts as a stopping condition in case of +URLs of increasing length, usually caused by a loop. +For more information about the default value +for this setting see: https://support.microsoft.com/en-us/topic/maximum-url-length-is-2-083-characters-in-internet-explorer-174e7c8a-6666-f4e0-6fd6-908b53c12246 .. setting:: USER_AGENT diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst index 73bedf655..a0a7b1fb6 100644 --- a/docs/topics/spider-middleware.rst +++ b/docs/topics/spider-middleware.rst @@ -440,4 +440,5 @@ UrlLengthMiddleware settings (see the settings documentation for more info): * :setting:`URLLENGTH_LIMIT` - The maximum URL length to allow for crawled URLs. + If ``0``, then :class:`~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware` is disabled.