mirror of https://github.com/scrapy/scrapy.git
Remove docs mentioning deprecated spider attrs. (#7176)
This commit is contained in:
parent
d8583a89c7
commit
588f3d4f65
|
|
@ -163,8 +163,8 @@ information on which commands must be run from inside projects, and which not.
|
||||||
|
|
||||||
Also keep in mind that some commands may have slightly different behaviours
|
Also keep in mind that some commands may have slightly different behaviours
|
||||||
when running them from inside projects. For example, the fetch command will use
|
when running them from inside projects. For example, the fetch command will use
|
||||||
spider-overridden behaviours (such as the ``user_agent`` attribute to override
|
spider-overridden behaviours (such as the ``custom_settings`` attribute to
|
||||||
the user-agent) if the url being fetched is associated with some specific
|
override settings) if the url being fetched is associated with some specific
|
||||||
spider. This is intentional, as the ``fetch`` command is meant to be used to
|
spider. This is intentional, as the ``fetch`` command is meant to be used to
|
||||||
check how spiders are downloading pages.
|
check how spiders are downloading pages.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -290,13 +290,12 @@ DownloadTimeoutMiddleware
|
||||||
.. class:: DownloadTimeoutMiddleware
|
.. class:: DownloadTimeoutMiddleware
|
||||||
|
|
||||||
This middleware sets the download timeout for requests specified in the
|
This middleware sets the download timeout for requests specified in the
|
||||||
:setting:`DOWNLOAD_TIMEOUT` setting or :attr:`download_timeout`
|
:setting:`DOWNLOAD_TIMEOUT` setting.
|
||||||
spider attribute.
|
|
||||||
|
|
||||||
.. note::
|
.. note::
|
||||||
|
|
||||||
You can also set download timeout per-request using
|
You can also set download timeout per-request using the
|
||||||
:reqmeta:`download_timeout` Request.meta key; this is supported
|
:reqmeta:`download_timeout` :attr:`.Request.meta` key; this is supported
|
||||||
even when DownloadTimeoutMiddleware is disabled.
|
even when DownloadTimeoutMiddleware is disabled.
|
||||||
|
|
||||||
HttpAuthMiddleware
|
HttpAuthMiddleware
|
||||||
|
|
@ -1212,9 +1211,8 @@ UserAgentMiddleware
|
||||||
|
|
||||||
.. class:: UserAgentMiddleware
|
.. class:: UserAgentMiddleware
|
||||||
|
|
||||||
Middleware that allows spiders to override the default user agent.
|
Middleware that sets the ``User-Agent`` header.
|
||||||
|
|
||||||
In order for a spider to override the default user agent, its ``user_agent``
|
The header value is taken from the :setting:`USER_AGENT` setting.
|
||||||
attribute must be set.
|
|
||||||
|
|
||||||
.. _DBM: https://en.wikipedia.org/wiki/Dbm
|
.. _DBM: https://en.wikipedia.org/wiki/Dbm
|
||||||
|
|
|
||||||
|
|
@ -1007,9 +1007,8 @@ The amount of time (in secs) that the downloader will wait before timing out.
|
||||||
|
|
||||||
.. note::
|
.. note::
|
||||||
|
|
||||||
This timeout can be set per spider using :attr:`download_timeout`
|
This timeout can be per-request using the :reqmeta:`download_timeout`
|
||||||
spider attribute and per-request using :reqmeta:`download_timeout`
|
:attr:`.Request.meta` key.
|
||||||
Request.meta key.
|
|
||||||
|
|
||||||
.. setting:: DOWNLOAD_MAXSIZE
|
.. setting:: DOWNLOAD_MAXSIZE
|
||||||
.. reqmeta:: download_maxsize
|
.. reqmeta:: download_maxsize
|
||||||
|
|
@ -1028,9 +1027,10 @@ ignored.
|
||||||
|
|
||||||
Use ``0`` to disable this limit.
|
Use ``0`` to disable this limit.
|
||||||
|
|
||||||
This limit can be set per spider using the :attr:`download_maxsize` spider
|
.. note::
|
||||||
attribute and per request using the :reqmeta:`download_maxsize` Request.meta
|
|
||||||
key.
|
This limit can be set per-request using the :reqmeta:`download_maxsize`
|
||||||
|
:attr:`.Request.meta` key.
|
||||||
|
|
||||||
.. setting:: DOWNLOAD_WARNSIZE
|
.. setting:: DOWNLOAD_WARNSIZE
|
||||||
.. reqmeta:: download_warnsize
|
.. reqmeta:: download_warnsize
|
||||||
|
|
@ -1045,9 +1045,10 @@ warning will be logged about it.
|
||||||
|
|
||||||
Use ``0`` to disable this limit.
|
Use ``0`` to disable this limit.
|
||||||
|
|
||||||
This limit can be set per spider using the :attr:`download_warnsize` spider
|
.. note::
|
||||||
attribute and per request using the :reqmeta:`download_warnsize` Request.meta
|
|
||||||
key.
|
This limit can be set per-request using the :reqmeta:`download_warnsize`
|
||||||
|
:attr:`.Request.meta` key.
|
||||||
|
|
||||||
.. setting:: DOWNLOAD_FAIL_ON_DATALOSS
|
.. setting:: DOWNLOAD_FAIL_ON_DATALOSS
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -355,11 +355,9 @@ Otherwise, you would cause iteration over a ``start_urls`` string
|
||||||
resulting in each character being seen as a separate url.
|
resulting in each character being seen as a separate url.
|
||||||
|
|
||||||
A valid use case is to set the http auth credentials
|
A valid use case is to set the http auth credentials
|
||||||
used by :class:`~scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware`
|
used by :class:`~scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware`::
|
||||||
or the user agent
|
|
||||||
used by :class:`~scrapy.downloadermiddlewares.useragent.UserAgentMiddleware`::
|
|
||||||
|
|
||||||
scrapy crawl myspider -a http_user=myuser -a http_pass=mypassword -a user_agent=mybot
|
scrapy crawl myspider -a http_user=myuser -a http_pass=mypassword
|
||||||
|
|
||||||
Spider arguments can also be passed through the Scrapyd ``schedule.json`` API.
|
Spider arguments can also be passed through the Scrapyd ``schedule.json`` API.
|
||||||
See `Scrapyd documentation`_.
|
See `Scrapyd documentation`_.
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue