mirror of https://github.com/scrapy/scrapy.git
Remove outdated version marker directives (< 2.8.0) from documentation (#7198)
This commit is contained in:
parent
180ca39b23
commit
9381ad893d
|
|
@ -4,8 +4,6 @@
|
||||||
asyncio
|
asyncio
|
||||||
=======
|
=======
|
||||||
|
|
||||||
.. versionadded:: 2.0
|
|
||||||
|
|
||||||
Scrapy has partial support for :mod:`asyncio`. After you :ref:`install the
|
Scrapy has partial support for :mod:`asyncio`. After you :ref:`install the
|
||||||
asyncio reactor <install-asyncio>`, you may use :mod:`asyncio` and
|
asyncio reactor <install-asyncio>`, you may use :mod:`asyncio` and
|
||||||
:mod:`asyncio`-powered libraries in any :doc:`coroutine <coroutines>`.
|
:mod:`asyncio`-powered libraries in any :doc:`coroutine <coroutines>`.
|
||||||
|
|
|
||||||
|
|
@ -233,9 +233,6 @@ genspider
|
||||||
* Syntax: ``scrapy genspider [-t template] <name> <domain or URL>``
|
* Syntax: ``scrapy genspider [-t template] <name> <domain or URL>``
|
||||||
* Requires project: *no*
|
* Requires project: *no*
|
||||||
|
|
||||||
.. versionadded:: 2.6.0
|
|
||||||
The ability to pass a URL instead of a domain.
|
|
||||||
|
|
||||||
Creates a new spider in the current folder or in the current project's ``spiders`` folder, if called from inside a project. The ``<name>`` parameter is set as the spider's ``name``, while ``<domain or URL>`` is used to generate the ``allowed_domains`` and ``start_urls`` spider's attributes.
|
Creates a new spider in the current folder or in the current project's ``spiders`` folder, if called from inside a project. The ``<name>`` parameter is set as the spider's ``name``, while ``<domain or URL>`` is used to generate the ``allowed_domains`` and ``start_urls`` spider's attributes.
|
||||||
|
|
||||||
Usage example::
|
Usage example::
|
||||||
|
|
@ -509,8 +506,6 @@ Supported options:
|
||||||
|
|
||||||
* ``--output`` or ``-o``: dump scraped items to a file
|
* ``--output`` or ``-o``: dump scraped items to a file
|
||||||
|
|
||||||
.. versionadded:: 2.3
|
|
||||||
|
|
||||||
.. skip: start
|
.. skip: start
|
||||||
|
|
||||||
Usage example::
|
Usage example::
|
||||||
|
|
|
||||||
|
|
@ -4,8 +4,6 @@
|
||||||
Coroutines
|
Coroutines
|
||||||
==========
|
==========
|
||||||
|
|
||||||
.. versionadded:: 2.0
|
|
||||||
|
|
||||||
Scrapy :ref:`supports <coroutine-support>` the :ref:`coroutine syntax <async>`
|
Scrapy :ref:`supports <coroutine-support>` the :ref:`coroutine syntax <async>`
|
||||||
(i.e. ``async def``).
|
(i.e. ``async def``).
|
||||||
|
|
||||||
|
|
@ -28,10 +26,6 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):
|
||||||
If you are using any custom or third-party :ref:`spider middleware
|
If you are using any custom or third-party :ref:`spider middleware
|
||||||
<topics-spider-middleware>`, see :ref:`sync-async-spider-middleware`.
|
<topics-spider-middleware>`, see :ref:`sync-async-spider-middleware`.
|
||||||
|
|
||||||
.. versionchanged:: 2.7
|
|
||||||
Output of async callbacks is now processed asynchronously instead of
|
|
||||||
collecting all of it first.
|
|
||||||
|
|
||||||
- The :meth:`process_item` method of
|
- The :meth:`process_item` method of
|
||||||
:ref:`item pipelines <topics-item-pipeline>`.
|
:ref:`item pipelines <topics-item-pipeline>`.
|
||||||
|
|
||||||
|
|
@ -53,8 +47,6 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):
|
||||||
See also :ref:`sync-async-spider-middleware` and
|
See also :ref:`sync-async-spider-middleware` and
|
||||||
:ref:`universal-spider-middleware`.
|
:ref:`universal-spider-middleware`.
|
||||||
|
|
||||||
.. versionadded:: 2.7
|
|
||||||
|
|
||||||
- The :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start` method
|
- The :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start` method
|
||||||
of :ref:`spider middlewares <custom-spider-middleware>`, which *must* be
|
of :ref:`spider middlewares <custom-spider-middleware>`, which *must* be
|
||||||
defined as an :term:`asynchronous generator`.
|
defined as an :term:`asynchronous generator`.
|
||||||
|
|
@ -307,8 +299,6 @@ You can also send multiple requests in parallel:
|
||||||
Mixing synchronous and asynchronous spider middlewares
|
Mixing synchronous and asynchronous spider middlewares
|
||||||
======================================================
|
======================================================
|
||||||
|
|
||||||
.. versionadded:: 2.7
|
|
||||||
|
|
||||||
The output of a :class:`~scrapy.Request` callback is passed as the ``result``
|
The output of a :class:`~scrapy.Request` callback is passed as the ``result``
|
||||||
parameter to the
|
parameter to the
|
||||||
:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output` method
|
:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output` method
|
||||||
|
|
@ -396,8 +386,6 @@ option. Otherwise, it's better to choose the second option.
|
||||||
Universal spider middlewares
|
Universal spider middlewares
|
||||||
----------------------------
|
----------------------------
|
||||||
|
|
||||||
.. versionadded:: 2.7
|
|
||||||
|
|
||||||
To allow writing a spider middleware that supports asynchronous execution of
|
To allow writing a spider middleware that supports asynchronous execution of
|
||||||
its ``process_spider_output`` method in Scrapy 2.7 and later (avoiding
|
its ``process_spider_output`` method in Scrapy 2.7 and later (avoiding
|
||||||
:ref:`asynchronous-to-synchronous conversions <sync-async-spider-middleware>`)
|
:ref:`asynchronous-to-synchronous conversions <sync-async-spider-middleware>`)
|
||||||
|
|
|
||||||
|
|
@ -917,10 +917,6 @@ Default: ``[]``
|
||||||
|
|
||||||
Meta tags within these tags are ignored.
|
Meta tags within these tags are ignored.
|
||||||
|
|
||||||
.. versionchanged:: 2.0
|
|
||||||
The default value of :setting:`METAREFRESH_IGNORE_TAGS` changed from
|
|
||||||
``["script", "noscript"]`` to ``[]``.
|
|
||||||
|
|
||||||
.. versionchanged:: 2.11.2
|
.. versionchanged:: 2.11.2
|
||||||
The default value of :setting:`METAREFRESH_IGNORE_TAGS` changed from
|
The default value of :setting:`METAREFRESH_IGNORE_TAGS` changed from
|
||||||
``[]`` to ``["noscript"]``.
|
``[]`` to ``["noscript"]``.
|
||||||
|
|
|
||||||
|
|
@ -83,8 +83,6 @@ This exception is raised to indicate an unsupported feature.
|
||||||
StopDownload
|
StopDownload
|
||||||
-------------
|
-------------
|
||||||
|
|
||||||
.. versionadded:: 2.2
|
|
||||||
|
|
||||||
.. exception:: StopDownload(fail=True)
|
.. exception:: StopDownload(fail=True)
|
||||||
|
|
||||||
Raised from a :class:`~scrapy.signals.bytes_received` or :class:`~scrapy.signals.headers_received`
|
Raised from a :class:`~scrapy.signals.bytes_received` or :class:`~scrapy.signals.headers_received`
|
||||||
|
|
|
||||||
|
|
@ -163,9 +163,6 @@ BaseItemExporter
|
||||||
populate their respective instance attributes: :attr:`fields_to_export`,
|
populate their respective instance attributes: :attr:`fields_to_export`,
|
||||||
:attr:`export_empty_fields`, :attr:`encoding`, :attr:`indent`.
|
:attr:`export_empty_fields`, :attr:`encoding`, :attr:`indent`.
|
||||||
|
|
||||||
.. versionadded:: 2.0
|
|
||||||
The *dont_fail* parameter.
|
|
||||||
|
|
||||||
.. method:: export_item(item)
|
.. method:: export_item(item)
|
||||||
|
|
||||||
Exports the given item. This method must be implemented in subclasses.
|
Exports the given item. This method must be implemented in subclasses.
|
||||||
|
|
|
||||||
|
|
@ -236,8 +236,6 @@ This storage backend uses :ref:`delayed file delivery <delayed-file-delivery>`.
|
||||||
Google Cloud Storage (GCS)
|
Google Cloud Storage (GCS)
|
||||||
--------------------------
|
--------------------------
|
||||||
|
|
||||||
.. versionadded:: 2.3
|
|
||||||
|
|
||||||
The feeds are stored on `Google Cloud Storage`_.
|
The feeds are stored on `Google Cloud Storage`_.
|
||||||
|
|
||||||
- URI scheme: ``gs``
|
- URI scheme: ``gs``
|
||||||
|
|
@ -303,8 +301,6 @@ feed URI, allowing item delivery to start way before the end of the crawl.
|
||||||
Item filtering
|
Item filtering
|
||||||
==============
|
==============
|
||||||
|
|
||||||
.. versionadded:: 2.6.0
|
|
||||||
|
|
||||||
You can filter items that you want to allow for a particular feed by using the
|
You can filter items that you want to allow for a particular feed by using the
|
||||||
``item_classes`` option in :ref:`feeds options <feed-options>`. Only items of
|
``item_classes`` option in :ref:`feeds options <feed-options>`. Only items of
|
||||||
the specified types will be added to the feed.
|
the specified types will be added to the feed.
|
||||||
|
|
@ -344,8 +340,6 @@ ItemFilter
|
||||||
Post-Processing
|
Post-Processing
|
||||||
===============
|
===============
|
||||||
|
|
||||||
.. versionadded:: 2.6.0
|
|
||||||
|
|
||||||
Scrapy provides an option to activate plugins to post-process feeds before they are exported
|
Scrapy provides an option to activate plugins to post-process feeds before they are exported
|
||||||
to feed storages. In addition to using :ref:`builtin plugins <builtin-plugins>`, you
|
to feed storages. In addition to using :ref:`builtin plugins <builtin-plugins>`, you
|
||||||
can create your own :ref:`plugins <custom-plugins>`.
|
can create your own :ref:`plugins <custom-plugins>`.
|
||||||
|
|
@ -425,8 +419,6 @@ These are the settings used for configuring the feed exports:
|
||||||
FEEDS
|
FEEDS
|
||||||
-----
|
-----
|
||||||
|
|
||||||
.. versionadded:: 2.1
|
|
||||||
|
|
||||||
Default: ``{}``
|
Default: ``{}``
|
||||||
|
|
||||||
A dictionary in which every key is a feed URI (or a :class:`pathlib.Path`
|
A dictionary in which every key is a feed URI (or a :class:`pathlib.Path`
|
||||||
|
|
@ -479,8 +471,6 @@ as a fallback value if that key is not provided for a specific feed definition:
|
||||||
- ``batch_item_count``: falls back to
|
- ``batch_item_count``: falls back to
|
||||||
:setting:`FEED_EXPORT_BATCH_ITEM_COUNT`.
|
:setting:`FEED_EXPORT_BATCH_ITEM_COUNT`.
|
||||||
|
|
||||||
.. versionadded:: 2.3.0
|
|
||||||
|
|
||||||
- ``encoding``: falls back to :setting:`FEED_EXPORT_ENCODING`.
|
- ``encoding``: falls back to :setting:`FEED_EXPORT_ENCODING`.
|
||||||
|
|
||||||
- ``fields``: falls back to :setting:`FEED_EXPORT_FIELDS`.
|
- ``fields``: falls back to :setting:`FEED_EXPORT_FIELDS`.
|
||||||
|
|
@ -489,20 +479,14 @@ as a fallback value if that key is not provided for a specific feed definition:
|
||||||
|
|
||||||
If undefined or empty, all items are exported.
|
If undefined or empty, all items are exported.
|
||||||
|
|
||||||
.. versionadded:: 2.6.0
|
|
||||||
|
|
||||||
- ``item_filter``: a :ref:`filter class <item-filter>` to filter items to export.
|
- ``item_filter``: a :ref:`filter class <item-filter>` to filter items to export.
|
||||||
|
|
||||||
:class:`~scrapy.extensions.feedexport.ItemFilter` is used be default.
|
:class:`~scrapy.extensions.feedexport.ItemFilter` is used be default.
|
||||||
|
|
||||||
.. versionadded:: 2.6.0
|
|
||||||
|
|
||||||
- ``indent``: falls back to :setting:`FEED_EXPORT_INDENT`.
|
- ``indent``: falls back to :setting:`FEED_EXPORT_INDENT`.
|
||||||
|
|
||||||
- ``item_export_kwargs``: :class:`dict` with keyword arguments for the corresponding :ref:`item exporter class <topics-exporters>`.
|
- ``item_export_kwargs``: :class:`dict` with keyword arguments for the corresponding :ref:`item exporter class <topics-exporters>`.
|
||||||
|
|
||||||
.. versionadded:: 2.4.0
|
|
||||||
|
|
||||||
- ``overwrite``: whether to overwrite the file if it already exists
|
- ``overwrite``: whether to overwrite the file if it already exists
|
||||||
(``True``) or append to its content (``False``).
|
(``True``) or append to its content (``False``).
|
||||||
|
|
||||||
|
|
@ -522,8 +506,6 @@ as a fallback value if that key is not provided for a specific feed definition:
|
||||||
|
|
||||||
- :ref:`topics-feed-storage-stdout`: ``False`` (overwriting is not supported)
|
- :ref:`topics-feed-storage-stdout`: ``False`` (overwriting is not supported)
|
||||||
|
|
||||||
.. versionadded:: 2.4.0
|
|
||||||
|
|
||||||
- ``store_empty``: falls back to :setting:`FEED_STORE_EMPTY`.
|
- ``store_empty``: falls back to :setting:`FEED_STORE_EMPTY`.
|
||||||
|
|
||||||
- ``uri_params``: falls back to :setting:`FEED_URI_PARAMS`.
|
- ``uri_params``: falls back to :setting:`FEED_URI_PARAMS`.
|
||||||
|
|
@ -532,8 +514,6 @@ as a fallback value if that key is not provided for a specific feed definition:
|
||||||
|
|
||||||
The plugins will be used in the order of the list passed.
|
The plugins will be used in the order of the list passed.
|
||||||
|
|
||||||
.. versionadded:: 2.6.0
|
|
||||||
|
|
||||||
.. setting:: FEED_EXPORT_ENCODING
|
.. setting:: FEED_EXPORT_ENCODING
|
||||||
|
|
||||||
FEED_EXPORT_ENCODING
|
FEED_EXPORT_ENCODING
|
||||||
|
|
@ -700,8 +680,6 @@ format in :setting:`FEED_EXPORTERS`. E.g., to disable the built-in CSV exporter
|
||||||
FEED_EXPORT_BATCH_ITEM_COUNT
|
FEED_EXPORT_BATCH_ITEM_COUNT
|
||||||
----------------------------
|
----------------------------
|
||||||
|
|
||||||
.. versionadded:: 2.3.0
|
|
||||||
|
|
||||||
Default: ``0``
|
Default: ``0``
|
||||||
|
|
||||||
If assigned an integer number higher than ``0``, Scrapy generates multiple output files
|
If assigned an integer number higher than ``0``, Scrapy generates multiple output files
|
||||||
|
|
@ -771,15 +749,11 @@ The function signature should be as follows:
|
||||||
If :setting:`FEED_EXPORT_BATCH_ITEM_COUNT` is ``0``, ``batch_id``
|
If :setting:`FEED_EXPORT_BATCH_ITEM_COUNT` is ``0``, ``batch_id``
|
||||||
is always ``1``.
|
is always ``1``.
|
||||||
|
|
||||||
.. versionadded:: 2.3.0
|
|
||||||
|
|
||||||
- ``batch_time``: UTC date and time, in ISO format with ``:``
|
- ``batch_time``: UTC date and time, in ISO format with ``:``
|
||||||
replaced with ``-``.
|
replaced with ``-``.
|
||||||
|
|
||||||
See :setting:`FEED_EXPORT_BATCH_ITEM_COUNT`.
|
See :setting:`FEED_EXPORT_BATCH_ITEM_COUNT`.
|
||||||
|
|
||||||
.. versionadded:: 2.3.0
|
|
||||||
|
|
||||||
- ``time``: ``batch_time``, with microseconds set to ``0``.
|
- ``time``: ``batch_time``, with microseconds set to ``0``.
|
||||||
:type params: dict
|
:type params: dict
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -80,8 +80,6 @@ Example:
|
||||||
Dataclass objects
|
Dataclass objects
|
||||||
-----------------
|
-----------------
|
||||||
|
|
||||||
.. versionadded:: 2.2
|
|
||||||
|
|
||||||
:func:`~dataclasses.dataclass` allows the defining of item classes with field names,
|
:func:`~dataclasses.dataclass` allows the defining of item classes with field names,
|
||||||
so that :ref:`item exporters <topics-exporters>` can export all fields by
|
so that :ref:`item exporters <topics-exporters>` can export all fields by
|
||||||
default even if the first scraped object does not have values for all of them.
|
default even if the first scraped object does not have values for all of them.
|
||||||
|
|
@ -112,8 +110,6 @@ Example:
|
||||||
attr.s objects
|
attr.s objects
|
||||||
--------------
|
--------------
|
||||||
|
|
||||||
.. versionadded:: 2.2
|
|
||||||
|
|
||||||
:func:`attr.s` allows the defining of item classes with field names,
|
:func:`attr.s` allows the defining of item classes with field names,
|
||||||
so that :ref:`item exporters <topics-exporters>` can export all fields by
|
so that :ref:`item exporters <topics-exporters>` can export all fields by
|
||||||
default even if the first scraped object does not have values for all of them.
|
default even if the first scraped object does not have values for all of them.
|
||||||
|
|
|
||||||
|
|
@ -76,10 +76,6 @@ LxmlLinkExtractor
|
||||||
If not given, it will default to
|
If not given, it will default to
|
||||||
:data:`scrapy.linkextractors.IGNORED_EXTENSIONS`.
|
:data:`scrapy.linkextractors.IGNORED_EXTENSIONS`.
|
||||||
|
|
||||||
.. versionchanged:: 2.0
|
|
||||||
:data:`~scrapy.linkextractors.IGNORED_EXTENSIONS` now includes
|
|
||||||
``7z``, ``7zip``, ``apk``, ``bz2``, ``cdr``, ``dmg``, ``ico``,
|
|
||||||
``iso``, ``tar``, ``tar.gz``, ``webm``, and ``xz``.
|
|
||||||
:type deny_extensions: list
|
:type deny_extensions: list
|
||||||
|
|
||||||
:param restrict_xpaths: is an XPath (or list of XPath's) which defines
|
:param restrict_xpaths: is an XPath (or list of XPath's) which defines
|
||||||
|
|
|
||||||
|
|
@ -174,9 +174,6 @@ with the data to be parsed, and return a parsed value. So you can use any
|
||||||
function as input or output processor. The only requirement is that they must
|
function as input or output processor. The only requirement is that they must
|
||||||
accept one (and only one) positional argument, which will be an iterable.
|
accept one (and only one) positional argument, which will be an iterable.
|
||||||
|
|
||||||
.. versionchanged:: 2.0
|
|
||||||
Processors no longer need to be methods.
|
|
||||||
|
|
||||||
.. note:: Both input and output processors must receive an iterable as their
|
.. note:: Both input and output processors must receive an iterable as their
|
||||||
first argument. The output of those functions can be anything. The result of
|
first argument. The output of those functions can be anything. The result of
|
||||||
input processors will be appended to an internal list (in the Loader)
|
input processors will be appended to an internal list (in the Loader)
|
||||||
|
|
|
||||||
|
|
@ -212,8 +212,6 @@ Where:
|
||||||
FTP server storage
|
FTP server storage
|
||||||
------------------
|
------------------
|
||||||
|
|
||||||
.. versionadded:: 2.0
|
|
||||||
|
|
||||||
:setting:`FILES_STORE` and :setting:`IMAGES_STORE` can point to an FTP server.
|
:setting:`FILES_STORE` and :setting:`IMAGES_STORE` can point to an FTP server.
|
||||||
Scrapy will automatically upload the files to the server.
|
Scrapy will automatically upload the files to the server.
|
||||||
|
|
||||||
|
|
@ -547,9 +545,6 @@ See here the methods that you can override in your custom Files Pipeline:
|
||||||
By default the :meth:`file_path` method returns
|
By default the :meth:`file_path` method returns
|
||||||
``full/<request URL hash>.<extension>``.
|
``full/<request URL hash>.<extension>``.
|
||||||
|
|
||||||
.. versionadded:: 2.4
|
|
||||||
The *item* parameter.
|
|
||||||
|
|
||||||
.. method:: FilesPipeline.get_media_requests(item, info)
|
.. method:: FilesPipeline.get_media_requests(item, info)
|
||||||
|
|
||||||
As seen on the workflow, the pipeline will get the URLs of the images to
|
As seen on the workflow, the pipeline will get the URLs of the images to
|
||||||
|
|
@ -590,8 +585,6 @@ See here the methods that you can override in your custom Files Pipeline:
|
||||||
|
|
||||||
* ``status`` - the file status indication.
|
* ``status`` - the file status indication.
|
||||||
|
|
||||||
.. versionadded:: 2.2
|
|
||||||
|
|
||||||
It can be one of the following:
|
It can be one of the following:
|
||||||
|
|
||||||
* ``downloaded`` - file was downloaded.
|
* ``downloaded`` - file was downloaded.
|
||||||
|
|
@ -705,9 +698,6 @@ See here the methods that you can override in your custom Images Pipeline:
|
||||||
By default the :meth:`file_path` method returns
|
By default the :meth:`file_path` method returns
|
||||||
``full/<request URL hash>.<extension>``.
|
``full/<request URL hash>.<extension>``.
|
||||||
|
|
||||||
.. versionadded:: 2.4
|
|
||||||
The *item* parameter.
|
|
||||||
|
|
||||||
.. method:: ImagesPipeline.thumb_path(self, request, thumb_id, response=None, info=None, *, item=None)
|
.. method:: ImagesPipeline.thumb_path(self, request, thumb_id, response=None, info=None, *, item=None)
|
||||||
|
|
||||||
This method is called for every item of :setting:`IMAGES_THUMBS` per downloaded item. It returns the
|
This method is called for every item of :setting:`IMAGES_THUMBS` per downloaded item. It returns the
|
||||||
|
|
|
||||||
|
|
@ -32,10 +32,6 @@ Request objects
|
||||||
:type url: str
|
:type url: str
|
||||||
|
|
||||||
:param callback: sets :attr:`callback`, defaults to ``None``.
|
:param callback: sets :attr:`callback`, defaults to ``None``.
|
||||||
|
|
||||||
.. versionchanged:: 2.0
|
|
||||||
The *callback* parameter is no longer required when the *errback*
|
|
||||||
parameter is specified.
|
|
||||||
:type callback: Callable[Concatenate[Response, ...], Any] | None
|
:type callback: Callable[Concatenate[Response, ...], Any] | None
|
||||||
|
|
||||||
:param method: the HTTP method of this request. Defaults to ``'GET'``.
|
:param method: the HTTP method of this request. Defaults to ``'GET'``.
|
||||||
|
|
@ -116,10 +112,6 @@ Request objects
|
||||||
:class:`scrapy.Request.cookies <scrapy.Request>` parameter. This is a known
|
:class:`scrapy.Request.cookies <scrapy.Request>` parameter. This is a known
|
||||||
current limitation that is being worked on.
|
current limitation that is being worked on.
|
||||||
|
|
||||||
.. versionadded:: 2.6.0
|
|
||||||
Cookie values that are :class:`bool`, :class:`float` or :class:`int`
|
|
||||||
are casted to :class:`str`.
|
|
||||||
|
|
||||||
:type cookies: dict or list
|
:type cookies: dict or list
|
||||||
|
|
||||||
:param encoding: the encoding of this request (defaults to ``'utf-8'``).
|
:param encoding: the encoding of this request (defaults to ``'utf-8'``).
|
||||||
|
|
@ -134,10 +126,6 @@ Request objects
|
||||||
:type dont_filter: bool
|
:type dont_filter: bool
|
||||||
|
|
||||||
:param errback: sets :attr:`errback`, defaults to ``None``.
|
:param errback: sets :attr:`errback`, defaults to ``None``.
|
||||||
|
|
||||||
.. versionchanged:: 2.0
|
|
||||||
The *callback* parameter is no longer required when the *errback*
|
|
||||||
parameter is specified.
|
|
||||||
:type errback: Callable[[Failure], Any] | None
|
:type errback: Callable[[Failure], Any] | None
|
||||||
|
|
||||||
:param flags: Flags sent to the request, can be used for logging or similar purposes.
|
:param flags: Flags sent to the request, can be used for logging or similar purposes.
|
||||||
|
|
@ -448,8 +436,6 @@ To change how request fingerprints are built for your requests, use the
|
||||||
REQUEST_FINGERPRINTER_CLASS
|
REQUEST_FINGERPRINTER_CLASS
|
||||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
.. versionadded:: 2.7
|
|
||||||
|
|
||||||
Default: :class:`scrapy.utils.request.RequestFingerprinter`
|
Default: :class:`scrapy.utils.request.RequestFingerprinter`
|
||||||
|
|
||||||
A :ref:`request fingerprinter class <custom-request-fingerprinter>` or its
|
A :ref:`request fingerprinter class <custom-request-fingerprinter>` or its
|
||||||
|
|
@ -988,15 +974,6 @@ Response objects
|
||||||
For instance: "HTTP/1.0", "HTTP/1.1", "h2"
|
For instance: "HTTP/1.0", "HTTP/1.1", "h2"
|
||||||
:type protocol: :class:`str`
|
:type protocol: :class:`str`
|
||||||
|
|
||||||
.. versionadded:: 2.0.0
|
|
||||||
The ``certificate`` parameter.
|
|
||||||
|
|
||||||
.. versionadded:: 2.1.0
|
|
||||||
The ``ip_address`` parameter.
|
|
||||||
|
|
||||||
.. versionadded:: 2.5.0
|
|
||||||
The ``protocol`` parameter.
|
|
||||||
|
|
||||||
.. attribute:: Response.url
|
.. attribute:: Response.url
|
||||||
|
|
||||||
A string containing the URL of the response.
|
A string containing the URL of the response.
|
||||||
|
|
@ -1062,8 +1039,6 @@ Response objects
|
||||||
|
|
||||||
.. attribute:: Response.cb_kwargs
|
.. attribute:: Response.cb_kwargs
|
||||||
|
|
||||||
.. versionadded:: 2.0
|
|
||||||
|
|
||||||
A shortcut to the :attr:`~scrapy.Request.cb_kwargs` attribute of the
|
A shortcut to the :attr:`~scrapy.Request.cb_kwargs` attribute of the
|
||||||
:attr:`Response.request` object (i.e. ``self.request.cb_kwargs``).
|
:attr:`Response.request` object (i.e. ``self.request.cb_kwargs``).
|
||||||
|
|
||||||
|
|
@ -1082,8 +1057,6 @@ Response objects
|
||||||
|
|
||||||
.. attribute:: Response.certificate
|
.. attribute:: Response.certificate
|
||||||
|
|
||||||
.. versionadded:: 2.0.0
|
|
||||||
|
|
||||||
A :class:`twisted.internet.ssl.Certificate` object representing
|
A :class:`twisted.internet.ssl.Certificate` object representing
|
||||||
the server's SSL certificate.
|
the server's SSL certificate.
|
||||||
|
|
||||||
|
|
@ -1091,8 +1064,6 @@ Response objects
|
||||||
|
|
||||||
.. attribute:: Response.ip_address
|
.. attribute:: Response.ip_address
|
||||||
|
|
||||||
.. versionadded:: 2.1.0
|
|
||||||
|
|
||||||
The IP address of the server from which the Response originated.
|
The IP address of the server from which the Response originated.
|
||||||
|
|
||||||
This attribute is currently only populated by the HTTP 1.1 download
|
This attribute is currently only populated by the HTTP 1.1 download
|
||||||
|
|
@ -1101,8 +1072,6 @@ Response objects
|
||||||
|
|
||||||
.. attribute:: Response.protocol
|
.. attribute:: Response.protocol
|
||||||
|
|
||||||
.. versionadded:: 2.5.0
|
|
||||||
|
|
||||||
The protocol that was used to download the response.
|
The protocol that was used to download the response.
|
||||||
For instance: "HTTP/1.0", "HTTP/1.1"
|
For instance: "HTTP/1.0", "HTTP/1.1"
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -182,8 +182,6 @@ Setting values must be :ref:`picklable <pickle-picklable>`.
|
||||||
Import paths and classes
|
Import paths and classes
|
||||||
========================
|
========================
|
||||||
|
|
||||||
.. versionadded:: 2.4.0
|
|
||||||
|
|
||||||
When a setting references a callable object to be imported by Scrapy, such as a
|
When a setting references a callable object to be imported by Scrapy, such as a
|
||||||
class or a function, there are two different ways you can specify that object:
|
class or a function, there are two different ways you can specify that object:
|
||||||
|
|
||||||
|
|
@ -667,8 +665,6 @@ DNS in-memory cache size.
|
||||||
DNS_RESOLVER
|
DNS_RESOLVER
|
||||||
------------
|
------------
|
||||||
|
|
||||||
.. versionadded:: 2.0
|
|
||||||
|
|
||||||
Default: ``'scrapy.resolver.CachingThreadedResolver'``
|
Default: ``'scrapy.resolver.CachingThreadedResolver'``
|
||||||
|
|
||||||
The class to be used to resolve DNS names. The default ``scrapy.resolver.CachingThreadedResolver``
|
The class to be used to resolve DNS names. The default ``scrapy.resolver.CachingThreadedResolver``
|
||||||
|
|
@ -1798,8 +1794,6 @@ For available choices, see :setting:`SCHEDULER_MEMORY_QUEUE`.
|
||||||
SCRAPER_SLOT_MAX_ACTIVE_SIZE
|
SCRAPER_SLOT_MAX_ACTIVE_SIZE
|
||||||
----------------------------
|
----------------------------
|
||||||
|
|
||||||
.. versionadded:: 2.0
|
|
||||||
|
|
||||||
Default: ``5_000_000``
|
Default: ``5_000_000``
|
||||||
|
|
||||||
Soft limit (in bytes) for response data being processed.
|
Soft limit (in bytes) for response data being processed.
|
||||||
|
|
@ -1974,8 +1968,6 @@ in the ``project`` subdirectory.
|
||||||
TWISTED_REACTOR
|
TWISTED_REACTOR
|
||||||
---------------
|
---------------
|
||||||
|
|
||||||
.. versionadded:: 2.0
|
|
||||||
|
|
||||||
Default: ``"twisted.internet.asyncioreactor.AsyncioSelectorReactor"``
|
Default: ``"twisted.internet.asyncioreactor.AsyncioSelectorReactor"``
|
||||||
|
|
||||||
Import path of a given :mod:`~twisted.internet.reactor`.
|
Import path of a given :mod:`~twisted.internet.reactor`.
|
||||||
|
|
@ -2068,11 +2060,6 @@ If this setting is set ``None``, Scrapy will use the existing reactor if one is
|
||||||
already installed, or install the default reactor defined by Twisted for the
|
already installed, or install the default reactor defined by Twisted for the
|
||||||
current platform.
|
current platform.
|
||||||
|
|
||||||
.. versionchanged:: 2.7
|
|
||||||
The :command:`startproject` command now sets this setting to
|
|
||||||
``twisted.internet.asyncioreactor.AsyncioSelectorReactor`` in the generated
|
|
||||||
``settings.py`` file.
|
|
||||||
|
|
||||||
.. versionchanged:: 2.13
|
.. versionchanged:: 2.13
|
||||||
The default value was changed from ``None`` to
|
The default value was changed from ``None`` to
|
||||||
``"twisted.internet.asyncioreactor.AsyncioSelectorReactor"``.
|
``"twisted.internet.asyncioreactor.AsyncioSelectorReactor"``.
|
||||||
|
|
|
||||||
|
|
@ -423,8 +423,6 @@ request_left_downloader
|
||||||
.. signal:: request_left_downloader
|
.. signal:: request_left_downloader
|
||||||
.. function:: request_left_downloader(request, spider)
|
.. function:: request_left_downloader(request, spider)
|
||||||
|
|
||||||
.. versionadded:: 2.0
|
|
||||||
|
|
||||||
Sent when a :class:`~scrapy.Request` leaves the downloader, even in case of
|
Sent when a :class:`~scrapy.Request` leaves the downloader, even in case of
|
||||||
failure.
|
failure.
|
||||||
|
|
||||||
|
|
@ -442,8 +440,6 @@ bytes_received
|
||||||
.. signal:: bytes_received
|
.. signal:: bytes_received
|
||||||
.. function:: bytes_received(data, request, spider)
|
.. function:: bytes_received(data, request, spider)
|
||||||
|
|
||||||
.. versionadded:: 2.2
|
|
||||||
|
|
||||||
Sent by the HTTP 1.1 and S3 download handlers when a group of bytes is
|
Sent by the HTTP 1.1 and S3 download handlers when a group of bytes is
|
||||||
received for a specific request. This signal might be fired multiple
|
received for a specific request. This signal might be fired multiple
|
||||||
times for the same request, with partial data each time. For instance,
|
times for the same request, with partial data each time. For instance,
|
||||||
|
|
@ -472,8 +468,6 @@ headers_received
|
||||||
.. signal:: headers_received
|
.. signal:: headers_received
|
||||||
.. function:: headers_received(headers, body_length, request, spider)
|
.. function:: headers_received(headers, body_length, request, spider)
|
||||||
|
|
||||||
.. versionadded:: 2.5
|
|
||||||
|
|
||||||
Sent by the HTTP 1.1 and S3 download handlers when the response headers are
|
Sent by the HTTP 1.1 and S3 download handlers when the response headers are
|
||||||
available for a given request, before downloading any additional content.
|
available for a given request, before downloading any additional content.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -125,10 +125,6 @@ one or more of these methods:
|
||||||
:class:`~scrapy.Request` objects and :ref:`item objects
|
:class:`~scrapy.Request` objects and :ref:`item objects
|
||||||
<topics-items>`.
|
<topics-items>`.
|
||||||
|
|
||||||
.. versionchanged:: 2.7
|
|
||||||
This method may be defined as an :term:`asynchronous generator`, in
|
|
||||||
which case ``result`` is an :term:`asynchronous iterable`.
|
|
||||||
|
|
||||||
Consider defining this method as an :term:`asynchronous generator`,
|
Consider defining this method as an :term:`asynchronous generator`,
|
||||||
which will be a requirement in a future version of Scrapy. However, if
|
which will be a requirement in a future version of Scrapy. However, if
|
||||||
you plan on sharing your spider middleware with other people, consider
|
you plan on sharing your spider middleware with other people, consider
|
||||||
|
|
@ -148,8 +144,6 @@ one or more of these methods:
|
||||||
.. method:: process_spider_output_async(response, result)
|
.. method:: process_spider_output_async(response, result)
|
||||||
:async:
|
:async:
|
||||||
|
|
||||||
.. versionadded:: 2.7
|
|
||||||
|
|
||||||
If defined, this method must be an :term:`asynchronous generator`,
|
If defined, this method must be an :term:`asynchronous generator`,
|
||||||
which will be called instead of :meth:`process_spider_output` if
|
which will be called instead of :meth:`process_spider_output` if
|
||||||
``result`` is an :term:`asynchronous iterable`.
|
``result`` is an :term:`asynchronous iterable`.
|
||||||
|
|
|
||||||
|
|
@ -547,9 +547,6 @@ Crawling rules
|
||||||
callbacks for new requests when writing :class:`CrawlSpider`-based spiders;
|
callbacks for new requests when writing :class:`CrawlSpider`-based spiders;
|
||||||
unexpected behaviour can occur otherwise.
|
unexpected behaviour can occur otherwise.
|
||||||
|
|
||||||
.. versionadded:: 2.0
|
|
||||||
The *errback* parameter.
|
|
||||||
|
|
||||||
CrawlSpider example
|
CrawlSpider example
|
||||||
~~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue