diff --git a/docs/topics/commands.rst b/docs/topics/commands.rst index 4994fe1d6..b8bf41402 100644 --- a/docs/topics/commands.rst +++ b/docs/topics/commands.rst @@ -163,8 +163,8 @@ information on which commands must be run from inside projects, and which not. Also keep in mind that some commands may have slightly different behaviours when running them from inside projects. For example, the fetch command will use -spider-overridden behaviours (such as the ``user_agent`` attribute to override -the user-agent) if the url being fetched is associated with some specific +spider-overridden behaviours (such as the ``custom_settings`` attribute to +override settings) if the url being fetched is associated with some specific spider. This is intentional, as the ``fetch`` command is meant to be used to check how spiders are downloading pages. diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index 6d24482ec..44c7ac17b 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -290,13 +290,12 @@ DownloadTimeoutMiddleware .. class:: DownloadTimeoutMiddleware This middleware sets the download timeout for requests specified in the - :setting:`DOWNLOAD_TIMEOUT` setting or :attr:`download_timeout` - spider attribute. + :setting:`DOWNLOAD_TIMEOUT` setting. .. note:: - You can also set download timeout per-request using - :reqmeta:`download_timeout` Request.meta key; this is supported + You can also set download timeout per-request using the + :reqmeta:`download_timeout` :attr:`.Request.meta` key; this is supported even when DownloadTimeoutMiddleware is disabled. HttpAuthMiddleware @@ -1212,9 +1211,8 @@ UserAgentMiddleware .. class:: UserAgentMiddleware - Middleware that allows spiders to override the default user agent. + Middleware that sets the ``User-Agent`` header. - In order for a spider to override the default user agent, its ``user_agent`` - attribute must be set. + The header value is taken from the :setting:`USER_AGENT` setting. .. _DBM: https://en.wikipedia.org/wiki/Dbm diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 48ba573d7..64e5af250 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -1007,9 +1007,8 @@ The amount of time (in secs) that the downloader will wait before timing out. .. note:: - This timeout can be set per spider using :attr:`download_timeout` - spider attribute and per-request using :reqmeta:`download_timeout` - Request.meta key. + This timeout can be per-request using the :reqmeta:`download_timeout` + :attr:`.Request.meta` key. .. setting:: DOWNLOAD_MAXSIZE .. reqmeta:: download_maxsize @@ -1028,9 +1027,10 @@ ignored. Use ``0`` to disable this limit. -This limit can be set per spider using the :attr:`download_maxsize` spider -attribute and per request using the :reqmeta:`download_maxsize` Request.meta -key. +.. note:: + + This limit can be set per-request using the :reqmeta:`download_maxsize` + :attr:`.Request.meta` key. .. setting:: DOWNLOAD_WARNSIZE .. reqmeta:: download_warnsize @@ -1045,9 +1045,10 @@ warning will be logged about it. Use ``0`` to disable this limit. -This limit can be set per spider using the :attr:`download_warnsize` spider -attribute and per request using the :reqmeta:`download_warnsize` Request.meta -key. +.. note:: + + This limit can be set per-request using the :reqmeta:`download_warnsize` + :attr:`.Request.meta` key. .. setting:: DOWNLOAD_FAIL_ON_DATALOSS diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index 0a83b248e..b4dfb3757 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -355,11 +355,9 @@ Otherwise, you would cause iteration over a ``start_urls`` string resulting in each character being seen as a separate url. A valid use case is to set the http auth credentials -used by :class:`~scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware` -or the user agent -used by :class:`~scrapy.downloadermiddlewares.useragent.UserAgentMiddleware`:: +used by :class:`~scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware`:: - scrapy crawl myspider -a http_user=myuser -a http_pass=mypassword -a user_agent=mybot + scrapy crawl myspider -a http_user=myuser -a http_pass=mypassword Spider arguments can also be passed through the Scrapyd ``schedule.json`` API. See `Scrapyd documentation`_.