From fbb1236fd6ee283414360f5209b3a569e112c8cb Mon Sep 17 00:00:00 2001 From: Deepanshu verma <73387559+iDeepverma@users.noreply.github.com> Date: Fri, 1 Oct 2021 18:46:11 +0530 Subject: [PATCH] Update docs/topics/settings.rst MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit added suggestion Co-authored-by: Adrián Chaves --- docs/topics/settings.rst | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 4d3ae20cc..ed8f1f105 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -1645,12 +1645,19 @@ Default: ``2083`` Scope: ``spidermiddlewares.urllength`` -The maximum URL length to allow for crawled URLs. You can set this to ``0`` -to disable :class:`~scrapy.spidermiddlewares.urllength.UrlLengthMiddleware` for working -with URLs longer than the default value. The default limit acts as a stopping condition in case of -URLs of increasing length, usually caused by a loop. -For more information about the default value -for this setting see: https://support.microsoft.com/en-us/topic/maximum-url-length-is-2-083-characters-in-internet-explorer-174e7c8a-6666-f4e0-6fd6-908b53c12246 +The maximum URL length to allow for crawled URLs. + +This setting can act as a stopping condition in case of URLs of ever-increasing +length, which may be caused for example by a programming error either in the +target server or in your code. See also :setting:`REDIRECT_MAX_TIMES` and +:setting:`DEPTH_LIMIT`. + +Use ``0`` to allow URLs of any length. + +The default value is copied from the `Microsoft Internet Explorer maximum URL +length`_, even though this setting exists for different reasons. + +.. _Microsoft Internet Explorer maximum URL length: https://support.microsoft.com/en-us/topic/maximum-url-length-is-2-083-characters-in-internet-explorer-174e7c8a-6666-f4e0-6fd6-908b53c12246 .. setting:: USER_AGENT