Improve the release notes

This commit is contained in:
Adrián Chaves 2025-03-12 09:46:00 +01:00
parent a8a596b287
commit 6a03bf241f
3 changed files with 52 additions and 52 deletions

View File

@ -10,87 +10,62 @@ Scrapy VERSION (unreleased)
Highlights:
- Replaced ``start_requests`` with :meth:`~scrapy.Spider.yield_seeds`
- Replaced ``start_requests`` (sync) with :meth:`~scrapy.Spider.yield_seeds`
(async)
Backward-incompatible changes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- The second parameter of
``scrapy.core.engine.ExecutionEngine.open_spider()``, ``start_requests``,
has been removed. The starting requests are determined by the ``spider``
parameter instead.
- In ``scrapy.core.spidermw.SpiderMiddlewareManager``,
``process_start_requests()`` has been replaced by ``process_seeds()``.
- ``scrapy.core.spidermw.SpiderMiddlewareManager.process_start_requests()``
has been replaced by
``scrapy.core.spidermw.SpiderMiddlewareManager.process_seeds()``.
- In ``scrapy.core.engine.ExecutionEngine``:
- ``scrapy.core.engine.Slot`` has been renamed to
``scrapy.core.engine._Slot`` and should not be used.
- The second parameter of ``open_spider()``, ``start_requests``, has been
removed. The starting requests are determined by the ``spider``
parameter instead (see :meth:`~scrapy.Spider.yield_seeds`).
- ``scrapy.core.engine.ExecutionEngine.slot`` has been renamed to
``scrapy.core.engine.ExecutionEngine._slot`` and should not be used.
- The ``slot`` attribute has been renamed to ``_slot`` and should not be
used.
- In ``scrapy.core.engine``, the ``Slot`` class has been renamed to ``_Slot``
and should not be used.
- The ``slot`` :ref:`telnet variable <telnet-vars>` has been removed.
Deprecations
~~~~~~~~~~~~
- ``scrapy.Spider.start_requests`` is deprecated, use
:meth:`~scrapy.Spider.yield_seeds` instead.
- The ``start_requests()`` method of :class:`~scrapy.Spider` is deprecated,
use :meth:`~scrapy.Spider.yield_seeds` instead, or both to maintain support
for lower Scrapy versions.
To make spiders compatible with older Scrapy versions while avoiding a
deprecation warning on Scrapy VERSION and higher, you can define both
methods. For example:
(:issue:`456`, :issue:`3477`, :issue:`4467`, :issue:`5627`)
.. code-block:: python
async def yield_seeds(self) -> AsyncIterator[Any]:
for seed in self.start_requests():
yield seed
def start_requests(self) -> Iterable[Request]:
yield Request(url="https://toscrape.com")
(:issue:`456`, :issue:`3237`, :issue:`5627`, …)
- The ``process_start_requests`` method of :ref:`spider middlewares
- The ``process_start_requests()`` method of :ref:`spider middlewares
<topics-spider-middleware>` is deprecated, use
:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_seed` instead.
:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_seeds` instead, or
both to maintain support for lower Scrapy versions.
Defining both methods is OK, e.g. to support older Scrapy versions, but
only :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_seeds` is
used by Scrapy VERSION and higher.
(:issue:`456`, :issue:`3237`, :issue:`5627`, …)
- The ``scrapy.spiders.init.InitSpider`` spider class is deprecated.
..
TODO: Update the related issues lists including #456 to include other
related issues.
(:issue:`456`, :issue:`3477`, :issue:`4467`, :issue:`5627`)
New features
~~~~~~~~~~~~
- You can now yield the start requests and items of a spider from the
:meth:`~scrapy.Spider.yield_seeds` asynchronous generator.
:meth:`~scrapy.Spider.yield_seeds` spider method and from the
:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_seeds` spider
middleware method, both asynchronous generators.
This makes it possible to use asynchronous code to generate those start
requests and items, e.g. reading them from a queue service or database
using an asynchronous client, without the need to use a workaround such as
yielding the start requests and items from a spider callback instead.
using an asynchronous client, without workarounds.
(:issue:`456`, :issue:`3237`, :issue:`5627`, …)
(:issue:`456`, :issue:`3477`, :issue:`4467`, :issue:`5627`)
- The new :setting:`SEEDING_POLICY` setting allows customizing how spider
start requests and items are consumed.
Additionally, related new settings have been added:
:setting:`SEEDING_INITIAL_TIMEOUT`, :setting:`SEEDING_TIMEOUT`.
(:issue:`456`, :issue:`3237`, :issue:`5627`, …)
.. _release-2.12.0:

View File

@ -75,7 +75,13 @@ one or more of these methods:
Iterate over the output of :meth:`~scrapy.Spider.yield_seeds` or that
of the :meth:`process_seeds` method of an earlier spider middleware,
overriding it.
overriding it. For example:
.. code-block:: python
async def process_seeds(self, seeds) -> AsyncIterator[Any]:
async for seed in seeds:
yield seed
You may yield :class:`~scrapy.Request` or :ref:`item <topics-items>`
objects, same as :meth:`~scrapy.Spider.yield_seeds`, from *seeds* or
@ -84,6 +90,15 @@ one or more of these methods:
As with :meth:`~scrapy.Spider.yield_seeds`, how this method is iterated
is controlled by :setting:`SEEDING_POLICY`.
To write spider middlewares that work on Scrapy versions lower than
VERSION, define also a synchronous ``process_start_requests()`` method
that returns an iterable. For example:
.. code-block:: python
def process_start_requests(self, seeds, spider) -> Iterable[Request]:
yield from seeds
.. method:: process_spider_input(response, spider)
This method is called for each response that goes through the spider
@ -143,6 +158,7 @@ one or more of these methods:
:type spider: :class:`~scrapy.Spider` object
.. method:: process_spider_output_async(response, result, spider)
:async:
.. versionadded:: 2.7

View File

@ -115,6 +115,15 @@ class Spider(object_ref):
Use :setting:`SEEDING_POLICY` to set how :meth:`yield_seeds` is
iterated.
To write spiders that work on Scrapy versions lower than VERSION,
define also a synchronous ``start_requests()`` method that returns an
iterable. For example:
.. code-block:: python
def start_requests(self):
yield Request("https://toscrape.com/")
"""
for seed in self.start_requests():
yield seed