mirror of https://github.com/scrapy/scrapy.git
Remove docs mentioning deprecated spider attrs. (#7176)
This commit is contained in:
parent
d8583a89c7
commit
588f3d4f65
|
|
@ -163,8 +163,8 @@ information on which commands must be run from inside projects, and which not.
|
|||
|
||||
Also keep in mind that some commands may have slightly different behaviours
|
||||
when running them from inside projects. For example, the fetch command will use
|
||||
spider-overridden behaviours (such as the ``user_agent`` attribute to override
|
||||
the user-agent) if the url being fetched is associated with some specific
|
||||
spider-overridden behaviours (such as the ``custom_settings`` attribute to
|
||||
override settings) if the url being fetched is associated with some specific
|
||||
spider. This is intentional, as the ``fetch`` command is meant to be used to
|
||||
check how spiders are downloading pages.
|
||||
|
||||
|
|
|
|||
|
|
@ -290,13 +290,12 @@ DownloadTimeoutMiddleware
|
|||
.. class:: DownloadTimeoutMiddleware
|
||||
|
||||
This middleware sets the download timeout for requests specified in the
|
||||
:setting:`DOWNLOAD_TIMEOUT` setting or :attr:`download_timeout`
|
||||
spider attribute.
|
||||
:setting:`DOWNLOAD_TIMEOUT` setting.
|
||||
|
||||
.. note::
|
||||
|
||||
You can also set download timeout per-request using
|
||||
:reqmeta:`download_timeout` Request.meta key; this is supported
|
||||
You can also set download timeout per-request using the
|
||||
:reqmeta:`download_timeout` :attr:`.Request.meta` key; this is supported
|
||||
even when DownloadTimeoutMiddleware is disabled.
|
||||
|
||||
HttpAuthMiddleware
|
||||
|
|
@ -1212,9 +1211,8 @@ UserAgentMiddleware
|
|||
|
||||
.. class:: UserAgentMiddleware
|
||||
|
||||
Middleware that allows spiders to override the default user agent.
|
||||
Middleware that sets the ``User-Agent`` header.
|
||||
|
||||
In order for a spider to override the default user agent, its ``user_agent``
|
||||
attribute must be set.
|
||||
The header value is taken from the :setting:`USER_AGENT` setting.
|
||||
|
||||
.. _DBM: https://en.wikipedia.org/wiki/Dbm
|
||||
|
|
|
|||
|
|
@ -1007,9 +1007,8 @@ The amount of time (in secs) that the downloader will wait before timing out.
|
|||
|
||||
.. note::
|
||||
|
||||
This timeout can be set per spider using :attr:`download_timeout`
|
||||
spider attribute and per-request using :reqmeta:`download_timeout`
|
||||
Request.meta key.
|
||||
This timeout can be per-request using the :reqmeta:`download_timeout`
|
||||
:attr:`.Request.meta` key.
|
||||
|
||||
.. setting:: DOWNLOAD_MAXSIZE
|
||||
.. reqmeta:: download_maxsize
|
||||
|
|
@ -1028,9 +1027,10 @@ ignored.
|
|||
|
||||
Use ``0`` to disable this limit.
|
||||
|
||||
This limit can be set per spider using the :attr:`download_maxsize` spider
|
||||
attribute and per request using the :reqmeta:`download_maxsize` Request.meta
|
||||
key.
|
||||
.. note::
|
||||
|
||||
This limit can be set per-request using the :reqmeta:`download_maxsize`
|
||||
:attr:`.Request.meta` key.
|
||||
|
||||
.. setting:: DOWNLOAD_WARNSIZE
|
||||
.. reqmeta:: download_warnsize
|
||||
|
|
@ -1045,9 +1045,10 @@ warning will be logged about it.
|
|||
|
||||
Use ``0`` to disable this limit.
|
||||
|
||||
This limit can be set per spider using the :attr:`download_warnsize` spider
|
||||
attribute and per request using the :reqmeta:`download_warnsize` Request.meta
|
||||
key.
|
||||
.. note::
|
||||
|
||||
This limit can be set per-request using the :reqmeta:`download_warnsize`
|
||||
:attr:`.Request.meta` key.
|
||||
|
||||
.. setting:: DOWNLOAD_FAIL_ON_DATALOSS
|
||||
|
||||
|
|
|
|||
|
|
@ -355,11 +355,9 @@ Otherwise, you would cause iteration over a ``start_urls`` string
|
|||
resulting in each character being seen as a separate url.
|
||||
|
||||
A valid use case is to set the http auth credentials
|
||||
used by :class:`~scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware`
|
||||
or the user agent
|
||||
used by :class:`~scrapy.downloadermiddlewares.useragent.UserAgentMiddleware`::
|
||||
used by :class:`~scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware`::
|
||||
|
||||
scrapy crawl myspider -a http_user=myuser -a http_pass=mypassword -a user_agent=mybot
|
||||
scrapy crawl myspider -a http_user=myuser -a http_pass=mypassword
|
||||
|
||||
Spider arguments can also be passed through the Scrapyd ``schedule.json`` API.
|
||||
See `Scrapyd documentation`_.
|
||||
|
|
|
|||
Loading…
Reference in New Issue