diff --git a/docs/news.rst b/docs/news.rst index 972e91b97..7afcbb3cf 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -10,87 +10,62 @@ Scrapy VERSION (unreleased) Highlights: -- Replaced ``start_requests`` with :meth:`~scrapy.Spider.yield_seeds` +- Replaced ``start_requests`` (sync) with :meth:`~scrapy.Spider.yield_seeds` + (async) Backward-incompatible changes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- The second parameter of - ``scrapy.core.engine.ExecutionEngine.open_spider()``, ``start_requests``, - has been removed. The starting requests are determined by the ``spider`` - parameter instead. +- In ``scrapy.core.spidermw.SpiderMiddlewareManager``, + ``process_start_requests()`` has been replaced by ``process_seeds()``. -- ``scrapy.core.spidermw.SpiderMiddlewareManager.process_start_requests()`` - has been replaced by - ``scrapy.core.spidermw.SpiderMiddlewareManager.process_seeds()``. +- In ``scrapy.core.engine.ExecutionEngine``: -- ``scrapy.core.engine.Slot`` has been renamed to - ``scrapy.core.engine._Slot`` and should not be used. + - The second parameter of ``open_spider()``, ``start_requests``, has been + removed. The starting requests are determined by the ``spider`` + parameter instead (see :meth:`~scrapy.Spider.yield_seeds`). -- ``scrapy.core.engine.ExecutionEngine.slot`` has been renamed to - ``scrapy.core.engine.ExecutionEngine._slot`` and should not be used. + - The ``slot`` attribute has been renamed to ``_slot`` and should not be + used. + +- In ``scrapy.core.engine``, the ``Slot`` class has been renamed to ``_Slot`` + and should not be used. - The ``slot`` :ref:`telnet variable ` has been removed. Deprecations ~~~~~~~~~~~~ -- ``scrapy.Spider.start_requests`` is deprecated, use - :meth:`~scrapy.Spider.yield_seeds` instead. +- The ``start_requests()`` method of :class:`~scrapy.Spider` is deprecated, + use :meth:`~scrapy.Spider.yield_seeds` instead, or both to maintain support + for lower Scrapy versions. - To make spiders compatible with older Scrapy versions while avoiding a - deprecation warning on Scrapy VERSION and higher, you can define both - methods. For example: + (:issue:`456`, :issue:`3477`, :issue:`4467`, :issue:`5627`) - .. code-block:: python - - async def yield_seeds(self) -> AsyncIterator[Any]: - for seed in self.start_requests(): - yield seed - - - def start_requests(self) -> Iterable[Request]: - yield Request(url="https://toscrape.com") - - (:issue:`456`, :issue:`3237`, :issue:`5627`, …) - -- The ``process_start_requests`` method of :ref:`spider middlewares +- The ``process_start_requests()`` method of :ref:`spider middlewares ` is deprecated, use - :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_seed` instead. + :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_seeds` instead, or + both to maintain support for lower Scrapy versions. - Defining both methods is OK, e.g. to support older Scrapy versions, but - only :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_seeds` is - used by Scrapy VERSION and higher. - - (:issue:`456`, :issue:`3237`, :issue:`5627`, …) - -- The ``scrapy.spiders.init.InitSpider`` spider class is deprecated. - -.. - TODO: Update the related issues lists including #456 to include other - related issues. + (:issue:`456`, :issue:`3477`, :issue:`4467`, :issue:`5627`) New features ~~~~~~~~~~~~ - You can now yield the start requests and items of a spider from the - :meth:`~scrapy.Spider.yield_seeds` asynchronous generator. + :meth:`~scrapy.Spider.yield_seeds` spider method and from the + :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_seeds` spider + middleware method, both asynchronous generators. This makes it possible to use asynchronous code to generate those start requests and items, e.g. reading them from a queue service or database - using an asynchronous client, without the need to use a workaround such as - yielding the start requests and items from a spider callback instead. + using an asynchronous client, without workarounds. - (:issue:`456`, :issue:`3237`, :issue:`5627`, …) + (:issue:`456`, :issue:`3477`, :issue:`4467`, :issue:`5627`) - The new :setting:`SEEDING_POLICY` setting allows customizing how spider start requests and items are consumed. - Additionally, related new settings have been added: - :setting:`SEEDING_INITIAL_TIMEOUT`, :setting:`SEEDING_TIMEOUT`. - - (:issue:`456`, :issue:`3237`, :issue:`5627`, …) - .. _release-2.12.0: diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst index fdb7c9eaf..0517a4b8e 100644 --- a/docs/topics/spider-middleware.rst +++ b/docs/topics/spider-middleware.rst @@ -75,7 +75,13 @@ one or more of these methods: Iterate over the output of :meth:`~scrapy.Spider.yield_seeds` or that of the :meth:`process_seeds` method of an earlier spider middleware, - overriding it. + overriding it. For example: + + .. code-block:: python + + async def process_seeds(self, seeds) -> AsyncIterator[Any]: + async for seed in seeds: + yield seed You may yield :class:`~scrapy.Request` or :ref:`item ` objects, same as :meth:`~scrapy.Spider.yield_seeds`, from *seeds* or @@ -84,6 +90,15 @@ one or more of these methods: As with :meth:`~scrapy.Spider.yield_seeds`, how this method is iterated is controlled by :setting:`SEEDING_POLICY`. + To write spider middlewares that work on Scrapy versions lower than + VERSION, define also a synchronous ``process_start_requests()`` method + that returns an iterable. For example: + + .. code-block:: python + + def process_start_requests(self, seeds, spider) -> Iterable[Request]: + yield from seeds + .. method:: process_spider_input(response, spider) This method is called for each response that goes through the spider @@ -143,6 +158,7 @@ one or more of these methods: :type spider: :class:`~scrapy.Spider` object .. method:: process_spider_output_async(response, result, spider) + :async: .. versionadded:: 2.7 diff --git a/scrapy/spiders/__init__.py b/scrapy/spiders/__init__.py index 85bda1bbb..a6ba14697 100644 --- a/scrapy/spiders/__init__.py +++ b/scrapy/spiders/__init__.py @@ -115,6 +115,15 @@ class Spider(object_ref): Use :setting:`SEEDING_POLICY` to set how :meth:`yield_seeds` is iterated. + + To write spiders that work on Scrapy versions lower than VERSION, + define also a synchronous ``start_requests()`` method that returns an + iterable. For example: + + .. code-block:: python + + def start_requests(self): + yield Request("https://toscrape.com/") """ for seed in self.start_requests(): yield seed