Remove support for sync process_spider_output(). (#7504)

* Remove support for sync process_spider_output().

* Slight check fix.

* Don't expect exceptions from calling process_spider_output().

* Remove dead code.

* Fix test_deprecated_mw_spider_arg() to run all methods.

* Fix typing.

* Typos.

* Remove references to the removed section, move the universal section to the spider middleware docs and write it from a different perspective

* Fix indentation issue

* Minor rewordings

* older versions → lower versions

* Update news.rst

---------

Co-authored-by: Adrian Chaves <adrian@zyte.com>

docs/news.rst

@@ -1380,8 +1380,8 @@ Highlights:
 
 -   Added the :reqmeta:`allow_offsite` request meta key
 
--   :ref:`Spider middlewares that don't support asynchronous spider output
+-   Spider middlewares that don't support asynchronous spider output are
-    <sync-async-spider-middleware>` are deprecated
+    deprecated
 
 -   Added a base class for :ref:`universal spider middlewares
     <universal-spider-middleware>`
@@ -1519,13 +1519,11 @@ Deprecations
     ``start_queue_cls`` parameter.
     (:issue:`6752`)
 
--   :ref:`Spider middlewares that don't support asynchronous spider output
+-   Spider middlewares that don't support asynchronous spider output are
-    <sync-async-spider-middleware>` are deprecated. The async iterable
+    deprecated. The async iterable downgrading feature, needed for using such
-    downgrading feature, needed for using such middlewares with asynchronous
+    middlewares with asynchronous callbacks and with other spider middlewares
-    callbacks and with other spider middlewares that produce asynchronous
+    that produce asynchronous iterables, is also deprecated. Please update all
-    iterables, is also deprecated. Please update all such middlewares to
+    such middlewares to support asynchronous spider output. (:issue:`6664`)
-    support asynchronous spider output.
-    (:issue:`6664`)
 
 -   Functions that were imported from :mod:`w3lib.url` and re-exported in
     :mod:`scrapy.utils.url` are now deprecated, you should import them from
@@ -1784,9 +1782,8 @@ Documentation
 -   Documented the setting values set in the default project template.
     (:issue:`6762`, :issue:`6775`)
 
--   Improved the :ref:`docs <sync-async-spider-middleware>` about asynchronous
+-   Improved the docs about asynchronous iterable support in spider
-    iterable support in spider middlewares.
+    middlewares. (:issue:`6688`)
-    (:issue:`6688`)
 
 -   Improved the :ref:`docs <coroutine-deferred-apis>` about using
     :class:`~twisted.internet.defer.Deferred`-based APIs in coroutine-based

docs/topics/coroutines.rst

@@ -23,9 +23,6 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):
 
 -   :class:`~scrapy.Request` callbacks.
 
-    If you are using any custom or third-party :ref:`spider middleware
-    <topics-spider-middleware>`, see :ref:`sync-async-spider-middleware`.
-
 -   The :meth:`process_item` method of
     :ref:`item pipelines <topics-item-pipeline>`.
 
@@ -39,13 +36,9 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):
 
 -   The
     :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output`
-    method of :ref:`spider middlewares <topics-spider-middleware>`.
+    method of :ref:`spider middlewares <topics-spider-middleware>`, which
-
+    *must* be defined as an :term:`asynchronous generator` except in
-    If defined as a coroutine, it must be an :term:`asynchronous generator`.
+    :ref:`universal spider middlewares <universal-spider-middleware>`.
-    The input ``result`` parameter is an :term:`asynchronous iterable`.
-
-    See also :ref:`sync-async-spider-middleware` and
-    :ref:`universal-spider-middleware`.
 
 -   The :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start` method
     of :ref:`spider middlewares <custom-spider-middleware>`, which *must* be
@@ -277,136 +270,3 @@ You can also send multiple requests in parallel:
                 "price": responses[0][1].css(".price::text").get(),
                 "price2": responses[1][1].css(".color::text").get(),
             }
-
-
-.. _sync-async-spider-middleware:
-
-Mixing synchronous and asynchronous spider middlewares
-======================================================
-
-The output of a :class:`~scrapy.Request` callback is passed as the ``result``
-parameter to the
-:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output` method
-of the first :ref:`spider middleware <topics-spider-middleware>` from the
-:ref:`list of active spider middlewares <topics-spider-middleware-setting>`.
-Then the output of that ``process_spider_output`` method is passed to the
-``process_spider_output`` method of the next spider middleware, and so on for
-every active spider middleware.
-
-Scrapy supports mixing :ref:`coroutine methods <async>` and synchronous methods
-in this chain of calls.
-
-However, if any of the ``process_spider_output`` methods is defined as a
-synchronous method, and the previous ``Request`` callback or
-``process_spider_output`` method is a coroutine, there are some drawbacks to
-the asynchronous-to-synchronous conversion that Scrapy does so that the
-synchronous ``process_spider_output`` method gets a synchronous iterable as its
-``result`` parameter:
-
--   The whole output of the previous ``Request`` callback or
-    ``process_spider_output`` method is awaited at this point.
-
--   If an exception raises while awaiting the output of the previous
-    ``Request`` callback or ``process_spider_output`` method, none of that
-    output will be processed.
-
-    This contrasts with the regular behavior, where all items yielded before
-    an exception raises are processed.
-
-Asynchronous-to-synchronous conversions are supported for backward
-compatibility, but they are deprecated and will stop working in a future
-version of Scrapy.
-
-To avoid asynchronous-to-synchronous conversions, when defining ``Request``
-callbacks as coroutine methods or when using spider middlewares whose
-``process_spider_output`` method is an :term:`asynchronous generator`, all
-active spider middlewares must either have their ``process_spider_output``
-method defined as an asynchronous generator or :ref:`define a
-process_spider_output_async method <universal-spider-middleware>`.
-
-.. _sync-async-spider-middleware-users:
-
-For middleware users
---------------------
-
-If you have asynchronous callbacks or use asynchronous-only spider middlewares
-you should make sure the asynchronous-to-synchronous conversions
-:ref:`described above <sync-async-spider-middleware>` don't happen. To do this,
-make sure all spider middlewares you use support asynchronous spider output.
-Even if you don't have asynchronous callbacks and don't use asynchronous-only
-spider middlewares in your project, it's still a good idea to make sure all
-middlewares you use support asynchronous spider output, so that it will be easy
-to start using asynchronous callbacks in the future. Because of this, Scrapy
-logs a warning when it detects a synchronous-only spider middleware.
-
-If you want to update middlewares you wrote, see the :ref:`following section
-<sync-async-spider-middleware-authors>`. If you have 3rd-party middlewares that
-aren't yet updated by their authors, you can :ref:`subclass <tut-inheritance>`
-them to make them :ref:`universal <universal-spider-middleware>` and use the
-subclasses in your projects.
-
-.. _sync-async-spider-middleware-authors:
-
-For middleware authors
-----------------------
-
-If you have a spider middleware that defines a synchronous
-``process_spider_output`` method, you should update it to support asynchronous
-spider output for :ref:`better compatibility <sync-async-spider-middleware>`,
-even if you don't yet use it with asynchronous callbacks, especially if you
-publish this middleware for other people to use. You have two options for this:
-
-1. Make the middleware asynchronous, by making the ``process_spider_output``
-   method an :term:`asynchronous generator`.
-2. Make the middleware universal, as described in the :ref:`next section
-   <universal-spider-middleware>`.
-
-If your middleware won't be used in projects with synchronous-only middlewares,
-e.g. because it's an internal middleware and you know that all other
-middlewares in your projects are already updated, it's safe to choose the first
-option. Otherwise, it's better to choose the second option.
-
-.. _universal-spider-middleware:
-
-Universal spider middlewares
-----------------------------
-
-To allow writing a spider middleware that supports asynchronous execution of
-its ``process_spider_output`` method in Scrapy 2.7 and later (avoiding
-:ref:`asynchronous-to-synchronous conversions <sync-async-spider-middleware>`)
-while maintaining support for older Scrapy versions, you may define
-``process_spider_output`` as a synchronous method and define an
-:term:`asynchronous generator` version of that method with an alternative name:
-``process_spider_output_async``.
-
-For example:
-
-.. code-block:: python
-
-    class UniversalSpiderMiddleware:
-        def process_spider_output(self, response, result):
-            for r in result:
-                # ... do something with r
-                yield r
-
-        async def process_spider_output_async(self, response, result):
-            async for r in result:
-                # ... do something with r
-                yield r
-
-.. note:: This is an interim measure to allow, for a time, to write code that
-          works in Scrapy 2.7 and later without requiring
-          asynchronous-to-synchronous conversions, and works in earlier Scrapy
-          versions as well.
-
-          In some future version of Scrapy, however, this feature will be
-          deprecated and, eventually, in a later version of Scrapy, this
-          feature will be removed, and all spider middlewares will be expected
-          to define their ``process_spider_output`` method as an asynchronous
-          generator.
-
-Since 2.13.0, Scrapy provides a base class,
-:class:`~scrapy.spidermiddlewares.base.BaseSpiderMiddleware`, which implements
-the ``process_spider_output()`` and ``process_spider_output_async()`` methods,
-so instead of duplicating the processing code you can override the
-``get_processed_request()`` and/or the ``get_processed_item()`` method.

docs/topics/spider-middleware.rst

@@ -117,36 +117,28 @@ one or more of these methods:
         :type response: :class:`~scrapy.http.Response` object
 
     .. method:: process_spider_output(response, result)
+        :async:
 
-        This method is called with the results returned from the Spider, after
+        This method is an :term:`asynchronous generator` called with the
-        it has processed the response.
+        results from the spider after the spider has processed the response.
 
-        :meth:`process_spider_output` must return an iterable of
+        .. seealso:: :ref:`universal-spider-middleware`.
-        :class:`~scrapy.Request` objects and :ref:`item objects
-        <topics-items>`.
-
-        Consider defining this method as an :term:`asynchronous generator`,
-        which will be a requirement in a future version of Scrapy. However, if
-        you plan on sharing your spider middleware with other people, consider
-        either :ref:`enforcing Scrapy 2.7 <enforce-component-requirements>`
-        as a minimum requirement of your spider middleware, or :ref:`making
-        your spider middleware universal <universal-spider-middleware>` so that
-        it works with Scrapy versions earlier than Scrapy 2.7.
 
         :param response: the response which generated this output from the
           spider
         :type response: :class:`~scrapy.http.Response` object
 
-        :param result: the result returned by the spider
+        :param result: the results from the spider
-        :type result: an iterable of :class:`~scrapy.Request` objects and
+        :type result: an :term:`asynchronous iterable` of
-          :ref:`item objects <topics-items>`
+          :class:`~scrapy.Request` objects and :ref:`item objects
+          <topics-items>`
 
     .. method:: process_spider_output_async(response, result)
         :async:
 
-        If defined, this method must be an :term:`asynchronous generator`,
+        Alternative name for :meth:`process_spider_output` used when
-        which will be called instead of :meth:`process_spider_output` if
+        implementing a :ref:`universal spider middleware
-        ``result`` is an :term:`asynchronous iterable`.
+        <universal-spider-middleware>`.
 
     .. method:: process_spider_exception(response, exception)
 
@@ -174,13 +166,40 @@ one or more of these methods:
         :type exception: :exc:`Exception` object
 
 
+.. _universal-spider-middleware:
+
+Universal spider middlewares
+----------------------------
+
+In Scrapy 2.6.3 and lower, ``process_spider_output()`` must be a *synchronous*
+generator.
+
+To support those versions and higher Scrapy versions in the same middleware,
+rename your asynchronous :method:`~SpiderMiddleware.process_spider_output()`
+method to :method:`~SpiderMiddleware.process_spider_output_async()`, and define
+a synchronous ``process_spider_output()`` method to be used by 2.6.3 and lower
+versions.
+
+For example:
+
+.. code-block:: python
+
+    class UniversalSpiderMiddleware:
+        async def process_spider_output_async(self, response, result):
+            async for r in result:
+                # ... do something with r
+                yield r
+
+        def process_spider_output(self, response, result):
+            for r in result:
+                # ... do something with r
+                yield r
+
 Base class for custom spider middlewares
 ----------------------------------------
 
 Scrapy provides a base class for custom spider middlewares. It's not required
-to use it but it can help with simplifying middleware implementations and
+to use it but it can help with simplifying middleware implementations.
-reducing the amount of boilerplate code in :ref:`universal middlewares
-<universal-spider-middleware>`.
 
 .. module:: scrapy.spidermiddlewares.base
 

scrapy/core/spidermw.py

@@ -9,29 +9,28 @@ from __future__ import annotations
 import logging
 from collections.abc import AsyncIterator, Callable, Coroutine, Iterable
 from functools import wraps
-from inspect import isasyncgenfunction, iscoroutine
+from inspect import isasyncgenfunction
 from itertools import islice
 from typing import TYPE_CHECKING, Any, TypeAlias, TypeVar, cast
 from warnings import warn
 
-from twisted.internet.defer import Deferred, inlineCallbacks
 from twisted.python.failure import Failure
 
 from scrapy import Request, Spider
 from scrapy.exceptions import ScrapyDeprecationWarning, _InvalidOutput
 from scrapy.http import Response
 from scrapy.middleware import MiddlewareManager
-from scrapy.utils.asyncgen import as_async_generator, collect_asyncgen
+from scrapy.utils.asyncgen import as_async_generator
 from scrapy.utils.conf import build_component_list
 from scrapy.utils.defer import (
     _defer_sleep_async,
     deferred_from_coro,
     maybe_deferred_to_future,
 )
-from scrapy.utils.python import MutableAsyncChain, MutableChain, global_object_name
+from scrapy.utils.python import MutableAsyncChain, global_object_name
 
 if TYPE_CHECKING:
-    from collections.abc import Generator
+    from twisted.internet.defer import Deferred
 
     from scrapy.settings import BaseSettings
 
@@ -46,10 +45,6 @@ ScrapeFunc: TypeAlias = Callable[
 ]
 
 
-def _isiterable(o: Any) -> bool:
-    return isinstance(o, (Iterable, AsyncIterator))
-
-
 class SpiderMiddlewareManager(MiddlewareManager):
     component_name = "spider middleware"
 
@@ -67,13 +62,10 @@ class SpiderMiddlewareManager(MiddlewareManager):
         if hasattr(mw, "process_start"):
             self.methods["process_start"].appendleft(mw.process_start)
 
-        process_spider_output = self._get_async_method_pair(mw, "process_spider_output")
+        process_spider_output = self._get_process_spider_output(mw)
         self.methods["process_spider_output"].appendleft(process_spider_output)
-        if callable(process_spider_output):
+        if process_spider_output is not None:
             self._check_mw_method_spider_arg(process_spider_output)
-        elif isinstance(process_spider_output, tuple):
-            for m in process_spider_output:
-                self._check_mw_method_spider_arg(m)
 
         process_spider_exception = getattr(mw, "process_spider_exception", None)
         self.methods["process_spider_exception"].appendleft(process_spider_exception)
@@ -105,48 +97,7 @@ class SpiderMiddlewareManager(MiddlewareManager):
                 return await scrape_func(Failure(), request)
         return await scrape_func(response, request)
 
-    def _evaluate_iterable(
+    async def _evaluate_iterable(
-        self,
-        response: Response,
-        iterable: Iterable[_T] | AsyncIterator[_T],
-        exception_processor_index: int,
-        recover_to: MutableChain[_T] | MutableAsyncChain[_T],
-    ) -> Iterable[_T] | AsyncIterator[_T]:
-
-        if isinstance(iterable, AsyncIterator):
-            return self._process_async(
-                response,
-                iterable,
-                exception_processor_index,
-                cast("MutableAsyncChain[_T]", recover_to),
-            )
-        return self._process_sync(
-            response,
-            iterable,
-            exception_processor_index,
-            cast("MutableChain[_T]", recover_to),
-        )
-
-    def _process_sync(
-        self,
-        response: Response,
-        iterable: Iterable[_T],
-        exception_processor_index: int,
-        recover_to: MutableChain[_T],
-    ) -> Iterable[_T]:
-        try:
-            yield from iterable
-        except Exception as ex:
-            exception_result = cast(
-                "Failure | MutableChain[_T]",
-                self._process_spider_exception(response, ex, exception_processor_index),
-            )
-            if isinstance(exception_result, Failure):
-                raise
-            assert isinstance(recover_to, MutableChain)
-            recover_to.extend(exception_result)
-
-    async def _process_async(
         self,
         response: Response,
         iterable: AsyncIterator[_T],
@@ -157,13 +108,9 @@ class SpiderMiddlewareManager(MiddlewareManager):
             async for r in iterable:
                 yield r
         except Exception as ex:
-            exception_result = cast(
+            exception_result: MutableAsyncChain[_T] = self._process_spider_exception(
-                "Failure | MutableAsyncChain[_T]",
+                response, ex, exception_processor_index
-                self._process_spider_exception(response, ex, exception_processor_index),
             )
-            if isinstance(exception_result, Failure):
-                raise
-            assert isinstance(recover_to, MutableAsyncChain)
             recover_to.extend(exception_result)
 
     def _process_spider_exception(
@@ -171,7 +118,7 @@ class SpiderMiddlewareManager(MiddlewareManager):
         response: Response,
         exception: Exception,
         start_index: int = 0,
-    ) -> MutableChain[_T] | MutableAsyncChain[_T]:
+    ) -> MutableAsyncChain[_T]:
         # don't handle _InvalidOutput exception
         if isinstance(exception, _InvalidOutput):
             raise exception
@@ -181,28 +128,18 @@ class SpiderMiddlewareManager(MiddlewareManager):
         for method_index, method in enumerate(method_list, start=start_index):
             if method is None:
                 continue
-            method = cast("Callable", method)
             if method in self._mw_methods_requiring_spider:
                 result = method(
                     response=response, exception=exception, spider=self._spider
                 )
             else:
                 result = method(response=response, exception=exception)
-            if _isiterable(result):
+            if isinstance(result, (Iterable, AsyncIterator)):
                 # stop exception handling by handing control over to the
                 # process_spider_output chain if an iterable has been returned
-                dfd: Deferred[MutableChain[_T] | MutableAsyncChain[_T]] = (
+                if isinstance(result, Iterable):
-                    self._process_spider_output(response, result, method_index + 1)
+                    result = as_async_generator(result)
-                )
+                return self._process_spider_output(response, result, method_index + 1)
-                # _process_spider_output() returns a Deferred only because of downgrading so this can be
-                # simplified when downgrading is removed.
-                if dfd.called:
-                    # the result is available immediately if _process_spider_output didn't do downgrading
-                    return cast("MutableChain[_T] | MutableAsyncChain[_T]", dfd.result)
-                # we forbid waiting here because otherwise we would need to return a deferred from
-                # _process_spider_exception too, which complicates the architecture
-                msg = f"Async iterable returned from {global_object_name(method)} cannot be downgraded"
-                raise _InvalidOutput(msg)
             if result is None:
                 continue
             msg = (
@@ -212,124 +149,35 @@ class SpiderMiddlewareManager(MiddlewareManager):
             raise _InvalidOutput(msg)
         raise exception
 
-    # This method cannot be made async def, as _process_spider_exception relies on the Deferred result
+    def _process_spider_output(
-    # being available immediately which doesn't work when it's a wrapped coroutine.
-    # It also needs @inlineCallbacks only because of downgrading so it can be removed when downgrading is removed.
-    @inlineCallbacks
-    def _process_spider_output(  # noqa: PLR0912
         self,
         response: Response,
-        result: Iterable[_T] | AsyncIterator[_T],
+        result: AsyncIterator[_T],
         start_index: int = 0,
-    ) -> Generator[Deferred[Any], Any, MutableChain[_T] | MutableAsyncChain[_T]]:
+    ) -> MutableAsyncChain[_T]:
         # items in this iterable do not need to go through the process_spider_output
         # chain, they went through it already from the process_spider_exception method
-        recovered: MutableChain[_T] | MutableAsyncChain[_T]
+        recovered: MutableAsyncChain[_T] = MutableAsyncChain()
-        last_result_is_async = isinstance(result, AsyncIterator)
-        recovered = MutableAsyncChain() if last_result_is_async else MutableChain()
-
-        # There are three cases for the middleware: def foo, async def foo, def foo + async def foo_async.
-        # 1. def foo. Sync iterables are passed as is, async ones are downgraded.
-        # 2. async def foo. Sync iterables are upgraded, async ones are passed as is.
-        # 3. def foo + async def foo_async. Iterables are passed to the respective method.
-        # Storing methods and method tuples in the same list is weird but we should be able to roll this back
-        # when we drop this compatibility feature.
-
         method_list = islice(self.methods["process_spider_output"], start_index, None)
-        for method_index, method_pair in enumerate(method_list, start=start_index):
+        for method_index, method in enumerate(method_list, start=start_index):
-            if method_pair is None:
+            if method is None:
                 continue
-            need_upgrade = need_downgrade = False
+            if method in self._mw_methods_requiring_spider:
-            if isinstance(method_pair, tuple):
+                result = method(response=response, result=result, spider=self._spider)
-                # This tuple handling is only needed until _async compatibility methods are removed.
-                method_sync, method_async = method_pair
-                method = method_async if last_result_is_async else method_sync
             else:
-                method = method_pair
+                result = method(response=response, result=result)
-                if not last_result_is_async and isasyncgenfunction(method):
+            result = self._evaluate_iterable(
-                    need_upgrade = True
+                response, result, method_index + 1, recovered
-                elif last_result_is_async and not isasyncgenfunction(method):
+            )
-                    need_downgrade = True
+        return MutableAsyncChain(result, recovered)
-            try:
-                if need_upgrade:
-                    # Iterable -> AsyncIterator
-                    result = as_async_generator(result)
-                elif need_downgrade:
-                    logger.warning(
-                        f"Async iterable passed to {global_object_name(method)} was"
-                        f" downgraded to a non-async one. This is deprecated and will"
-                        f" stop working in a future version of Scrapy. Please see"
-                        f" https://docs.scrapy.org/en/latest/topics/coroutines.html#for-middleware-users"
-                        f" for more information."
-                    )
-                    assert isinstance(result, AsyncIterator)
-                    # AsyncIterator -> Iterable
-                    result = yield deferred_from_coro(collect_asyncgen(result))
-                    if isinstance(recovered, AsyncIterator):
-                        recovered_collected = yield deferred_from_coro(
-                            collect_asyncgen(recovered)
-                        )
-                        recovered = MutableChain(recovered_collected)
-                # might fail directly if the output value is not a generator
-                if method in self._mw_methods_requiring_spider:
-                    result = method(
-                        response=response, result=result, spider=self._spider
-                    )
-                else:
-                    result = method(response=response, result=result)
-            except Exception as ex:
-                exception_result: Failure | MutableChain[_T] | MutableAsyncChain[_T] = (
-                    self._process_spider_exception(response, ex, method_index + 1)
-                )
-                if isinstance(exception_result, Failure):
-                    raise
-                return exception_result
-            if _isiterable(result):
-                result = self._evaluate_iterable(
-                    response, result, method_index + 1, recovered
-                )
-            else:
-                if iscoroutine(result):
-                    result.close()  # Silence warning about not awaiting
-                    msg = (
-                        f"{global_object_name(method)} must be an asynchronous "
-                        f"generator (i.e. use yield)"
-                    )
-                else:
-                    msg = (
-                        f"{global_object_name(method)} must return an iterable, got "
-                        f"{type(result)}"
-                    )
-                raise _InvalidOutput(msg)
-            last_result_is_async = isinstance(result, AsyncIterator)
-
-        if last_result_is_async:
-            return MutableAsyncChain(result, recovered)
-        return MutableChain(result, recovered)  # type: ignore[arg-type]
 
     async def _process_callback_output(
-        self,
+        self, response: Response, result: AsyncIterator[_T]
-        response: Response,
+    ) -> MutableAsyncChain[_T]:
-        result: Iterable[_T] | AsyncIterator[_T],
+        recovered: MutableAsyncChain[_T] = MutableAsyncChain()
-    ) -> MutableChain[_T] | MutableAsyncChain[_T]:
-        recovered: MutableChain[_T] | MutableAsyncChain[_T]
-        if isinstance(result, AsyncIterator):
-            recovered = MutableAsyncChain()
-        else:
-            recovered = MutableChain()
         result = self._evaluate_iterable(response, result, 0, recovered)
-        result = await maybe_deferred_to_future(
+        result = self._process_spider_output(response, result)
-            cast(
+        return MutableAsyncChain(result, recovered)
-                "Deferred[Iterable[_T] | AsyncIterator[_T]]",
-                self._process_spider_output(response, result),
-            )
-        )
-        if isinstance(result, AsyncIterator):
-            return MutableAsyncChain(result, recovered)
-        if isinstance(recovered, AsyncIterator):
-            recovered_collected = await collect_asyncgen(recovered)
-            recovered = MutableChain(recovered_collected)
-        return MutableChain(result, recovered)
 
     def scrape_response(
         self,
@@ -340,7 +188,7 @@ class SpiderMiddlewareManager(MiddlewareManager):
         response: Response,
         request: Request,
         spider: Spider,
-    ) -> Deferred[MutableChain[_T] | MutableAsyncChain[_T]]:  # pragma: no cover
+    ) -> Deferred[MutableAsyncChain[_T]]:  # pragma: no cover
         warn(
             "SpiderMiddlewareManager.scrape_response() is deprecated, use scrape_response_async() instead",
             ScrapyDeprecationWarning,
@@ -363,7 +211,7 @@ class SpiderMiddlewareManager(MiddlewareManager):
         scrape_func: ScrapeFunc[_T],
         response: Response,
         request: Request,
-    ) -> MutableChain[_T] | MutableAsyncChain[_T]:
+    ) -> MutableAsyncChain[_T]:
         if not self.crawler:
             raise RuntimeError(
                 "scrape_response_async() called on a SpiderMiddlewareManager"
@@ -373,7 +221,8 @@ class SpiderMiddlewareManager(MiddlewareManager):
             it: Iterable[_T] | AsyncIterator[_T] = await self._process_spider_input(
                 scrape_func, response, request
             )
-            return await self._process_callback_output(response, it)
+            ait = it if isinstance(it, AsyncIterator) else as_async_generator(it)
+            return await self._process_callback_output(response, ait)
         except Exception as ex:
             await _defer_sleep_async()
             return self._process_spider_exception(response, ex)
@@ -399,27 +248,23 @@ class SpiderMiddlewareManager(MiddlewareManager):
 
     # This method is only needed until _async compatibility methods are removed.
     @staticmethod
-    def _get_async_method_pair(
+    def _get_process_spider_output(mw: Any) -> Callable | None:
-        mw: Any, methodname: str
+        normal_method: Callable | None = getattr(mw, "process_spider_output", None)
-    ) -> Callable | tuple[Callable, Callable] | None:
+        async_method: Callable | None = getattr(mw, "process_spider_output_async", None)
-        normal_method: Callable | None = getattr(mw, methodname, None)
-        methodname_async = methodname + "_async"
-        async_method: Callable | None = getattr(mw, methodname_async, None)
         if not async_method:
             if normal_method and not isasyncgenfunction(normal_method):
-                logger.warning(
+                raise TypeError(
                     f"Middleware {global_object_name(mw.__class__)} doesn't support"
-                    f" asynchronous spider output, this is deprecated and will stop"
+                    f" asynchronous spider output. Its process_spider_output() method"
-                    f" working in a future version of Scrapy. The middleware should"
+                    f" should be an async generator function or it should additionally"
-                    f" be updated to support it. Please see"
+                    f" define a process_spider_output_async() method."
-                    f" https://docs.scrapy.org/en/latest/topics/coroutines.html#for-middleware-users"
-                    f" for more information."
                 )
             return normal_method
         if not normal_method:
             logger.error(
-                f"Middleware {global_object_name(mw.__class__)} has {methodname_async} "
+                f"Middleware {global_object_name(mw.__class__)} has"
-                f"without {methodname}, skipping this method."
+                f" process_spider_output_async() without process_spider_output(),"
+                f" skipping this method. Please rename it to process_spider_output()."
             )
             return None
         if not isasyncgenfunction(async_method):
@@ -431,8 +276,8 @@ class SpiderMiddlewareManager(MiddlewareManager):
         if isasyncgenfunction(normal_method):
             logger.error(
                 f"{global_object_name(normal_method)} is an async "
-                f"generator function while {methodname_async} exists, "
+                f"generator function while process_spider_output_async() exists, "
-                f"skipping both methods."
+                f"skipping both methods. Please remove process_spider_output_async()."
             )
             return None
-        return normal_method, async_method
+        return async_method

scrapy/middleware.py

@@ -50,10 +50,7 @@ class MiddlewareManager(ABC):
             )
         self.middlewares: tuple[Any, ...] = middlewares
         # Only process_spider_output and process_spider_exception can be None.
-        # Only process_spider_output can be a tuple, and only until _async compatibility methods are removed.
+        self.methods: dict[str, deque[Callable | None]] = defaultdict(deque)
-        self.methods: dict[str, deque[Callable | tuple[Callable, Callable] | None]] = (
-            defaultdict(deque)
-        )
         self._mw_methods_requiring_spider: set[Callable] = set()
         for mw in middlewares:
             self._add_middleware(mw)

scrapy/utils/python.py

@@ -8,12 +8,14 @@ import gc
 import inspect
 import re
 import sys
+import warnings
 import weakref
 from collections.abc import AsyncIterator, Iterable, Mapping
 from functools import partial, wraps
 from itertools import chain
 from typing import TYPE_CHECKING, Any, Concatenate, ParamSpec, TypeVar, overload
 
+from scrapy.exceptions import ScrapyDeprecationWarning
 from scrapy.utils.asyncgen import as_async_generator
 
 if TYPE_CHECKING:
@@ -294,12 +296,13 @@ else:
         gc.collect()
 
 
-class MutableChain(Iterable[_T]):
+class MutableChain(Iterable[_T]):  # pragma: no cover
-    """
-    Thin wrapper around itertools.chain, allowing to add iterables "in-place"
-    """
-
     def __init__(self, *args: Iterable[_T]):
+        warnings.warn(
+            "MutableChain is deprecated and will be removed in a future Scrapy version.",
+            category=ScrapyDeprecationWarning,
+            stacklevel=2,
+        )
         self.data: Iterator[_T] = chain.from_iterable(args)
 
     def extend(self, *iterables: Iterable[_T]) -> None:

tests/test_request_cb_kwargs.py

@@ -38,8 +38,8 @@ class InjectArgumentsSpiderMiddleware:
         if request.callback.__name__ == "parse_spider_mw":
             request.cb_kwargs["from_process_spider_input"] = True
 
-    def process_spider_output(self, response, result):
+    async def process_spider_output(self, response, result):
-        for element in result:
+        async for element in result:
             if (
                 isinstance(element, Request)
                 and element.callback.__name__ == "parse_spider_mw_2"

tests/test_spidermiddleware.py

@@ -6,7 +6,6 @@ from typing import TYPE_CHECKING, Any, cast
 from unittest import mock
 
 import pytest
-from testfixtures import LogCapture
 from twisted.internet import defer
 
 from scrapy.core.spidermw import SpiderMiddlewareManager
@@ -63,20 +62,6 @@ class TestProcessSpiderInputInvalidOutput(TestSpiderMiddleware):
             await self._scrape_response()
 
 
-class TestProcessSpiderOutputInvalidOutput(TestSpiderMiddleware):
-    """Invalid return value for process_spider_output method"""
-
-    @coroutine_test
-    async def test_invalid_process_spider_output(self):
-        class InvalidProcessSpiderOutputMiddleware:
-            def process_spider_output(self, response, result):
-                return 1
-
-        self.mwman._add_middleware(InvalidProcessSpiderOutputMiddleware())
-        with pytest.raises(_InvalidOutput):
-            await self._scrape_response()
-
-
 class TestProcessSpiderExceptionInvalidOutput(TestSpiderMiddleware):
     """Invalid return value for process_spider_exception method"""
 
@@ -87,13 +72,15 @@ class TestProcessSpiderExceptionInvalidOutput(TestSpiderMiddleware):
                 return 1
 
         class RaiseExceptionProcessSpiderOutputMiddleware:
-            def process_spider_output(self, response, result):
+            async def process_spider_output(self, response, result):
                 raise RuntimeError
+                yield  # pylint: disable=unreachable
 
         self.mwman._add_middleware(InvalidProcessSpiderOutputExceptionMiddleware())
         self.mwman._add_middleware(RaiseExceptionProcessSpiderOutputMiddleware())
+        it = await self._scrape_response()
         with pytest.raises(_InvalidOutput):
-            await self._scrape_response()
+            await collect_asyncgen(it)
 
 
 class TestProcessSpiderExceptionReRaise(TestSpiderMiddleware):
@@ -106,13 +93,15 @@ class TestProcessSpiderExceptionReRaise(TestSpiderMiddleware):
                 return None
 
         class RaiseExceptionProcessSpiderOutputMiddleware:
-            def process_spider_output(self, response, result):
+            async def process_spider_output(self, response, result):
                 1 / 0
+                yield
 
         self.mwman._add_middleware(ProcessSpiderExceptionReturnNoneMiddleware())
         self.mwman._add_middleware(RaiseExceptionProcessSpiderOutputMiddleware())
+        it = await self._scrape_response()
         with pytest.raises(ZeroDivisionError):
-            await self._scrape_response()
+            await collect_asyncgen(it)
 
 
 class TestBaseAsyncSpiderMiddleware(TestSpiderMiddleware):
@@ -155,43 +144,19 @@ class TestBaseAsyncSpiderMiddleware(TestSpiderMiddleware):
             self._scrape_func, self.response, self.request
         )
 
-    async def _test_simple_base(
-        self,
-        *mw_classes: type[Any],
-        downgrade: bool = False,
-        start_index: int | None = None,
-    ) -> None:
-        with LogCapture() as log:
-            result = await self._get_middleware_result(
-                *mw_classes, start_index=start_index
-            )
-        assert isinstance(result, Iterable)
-        result_list = list(result)
-        assert len(result_list) == self.RESULT_COUNT
-        assert isinstance(result_list[0], self.ITEM_TYPE)
-        assert ("downgraded to a non-async" in str(log)) == downgrade
-        assert ("doesn't support asynchronous spider output" in str(log)) == (
-            ProcessSpiderOutputSimpleMiddleware in mw_classes
-        )
-
     async def _test_asyncgen_base(
         self,
         *mw_classes: type[Any],
-        downgrade: bool = False,
         start_index: int | None = None,
     ) -> None:
-        with LogCapture() as log:
+        result = await self._get_middleware_result(*mw_classes, start_index=start_index)
-            result = await self._get_middleware_result(
-                *mw_classes, start_index=start_index
-            )
         assert isinstance(result, AsyncIterator)
         result_list = await collect_asyncgen(result)
         assert len(result_list) == self.RESULT_COUNT
         assert isinstance(result_list[0], self.ITEM_TYPE)
-        assert ("downgraded to a non-async" in str(log)) == downgrade
 
 
-class ProcessSpiderOutputSimpleMiddleware:
+class ProcessSpiderOutputSyncMiddleware:
     def process_spider_output(self, response, result):
         yield from result
 
@@ -232,53 +197,36 @@ class TestProcessSpiderOutputSimple(TestBaseAsyncSpiderMiddleware):
     """process_spider_output tests for simple callbacks"""
 
     ITEM_TYPE = dict
-    MW_SIMPLE = ProcessSpiderOutputSimpleMiddleware
+    MW_SYNC = ProcessSpiderOutputSyncMiddleware
     MW_ASYNCGEN = ProcessSpiderOutputAsyncGenMiddleware
     MW_UNIVERSAL = ProcessSpiderOutputUniversalMiddleware
 
     @coroutine_test
-    async def test_simple(self):
+    async def test_sync(self):
-        """Simple mw"""
+        """Unsupported sync mw"""
-        await self._test_simple_base(self.MW_SIMPLE)
+        with pytest.raises(
+            TypeError, match=r"doesn't support asynchronous spider output"
+        ):
+            await self._get_middleware_result(self.MW_SYNC)
 
     @coroutine_test
     async def test_asyncgen(self):
-        """Asyncgen mw; upgrade"""
+        """Asyncgen mw"""
         await self._test_asyncgen_base(self.MW_ASYNCGEN)
 
-    @coroutine_test
-    async def test_simple_asyncgen(self):
-        """Simple mw -> asyncgen mw; upgrade"""
-        await self._test_asyncgen_base(self.MW_ASYNCGEN, self.MW_SIMPLE)
-
-    @coroutine_test
-    async def test_asyncgen_simple(self):
-        """Asyncgen mw -> simple mw; upgrade then downgrade"""
-        await self._test_simple_base(self.MW_SIMPLE, self.MW_ASYNCGEN, downgrade=True)
-
     @coroutine_test
     async def test_universal(self):
         """Universal mw"""
-        await self._test_simple_base(self.MW_UNIVERSAL)
+        await self._test_asyncgen_base(self.MW_UNIVERSAL)
-
-    @coroutine_test
-    async def test_universal_simple(self):
-        """Universal mw -> simple mw"""
-        await self._test_simple_base(self.MW_SIMPLE, self.MW_UNIVERSAL)
-
-    @coroutine_test
-    async def test_simple_universal(self):
-        """Simple mw -> universal mw"""
-        await self._test_simple_base(self.MW_UNIVERSAL, self.MW_SIMPLE)
 
     @coroutine_test
     async def test_universal_asyncgen(self):
-        """Universal mw -> asyncgen mw; upgrade"""
+        """Universal mw -> asyncgen mw"""
         await self._test_asyncgen_base(self.MW_ASYNCGEN, self.MW_UNIVERSAL)
 
     @coroutine_test
     async def test_asyncgen_universal(self):
-        """Asyncgen mw -> universal mw; upgrade"""
+        """Asyncgen mw -> universal mw"""
         await self._test_asyncgen_base(self.MW_UNIVERSAL, self.MW_ASYNCGEN)
 
 
@@ -289,31 +237,6 @@ class TestProcessSpiderOutputAsyncGen(TestProcessSpiderOutputSimple):
         for item in super()._callback():
             yield item
 
-    @coroutine_test
-    async def test_simple(self):
-        """Simple mw; downgrade"""
-        await self._test_simple_base(self.MW_SIMPLE, downgrade=True)
-
-    @coroutine_test
-    async def test_simple_asyncgen(self):
-        """Simple mw -> asyncgen mw; downgrade then upgrade"""
-        await self._test_asyncgen_base(self.MW_ASYNCGEN, self.MW_SIMPLE, downgrade=True)
-
-    @coroutine_test
-    async def test_universal(self):
-        """Universal mw"""
-        await self._test_asyncgen_base(self.MW_UNIVERSAL)
-
-    @coroutine_test
-    async def test_universal_simple(self):
-        """Universal mw -> simple mw; downgrade"""
-        await self._test_simple_base(self.MW_SIMPLE, self.MW_UNIVERSAL, downgrade=True)
-
-    @coroutine_test
-    async def test_simple_universal(self):
-        """Simple mw -> universal mw; downgrade"""
-        await self._test_simple_base(self.MW_UNIVERSAL, self.MW_SIMPLE, downgrade=True)
-
 
 class ProcessSpiderOutputNonIterableMiddleware:
     def process_spider_output(self, response, result):
@@ -325,24 +248,6 @@ class ProcessSpiderOutputCoroutineMiddleware:
         return result
 
 
-class TestProcessSpiderOutputInvalidResult(TestBaseAsyncSpiderMiddleware):
-    @coroutine_test
-    async def test_non_iterable(self):
-        with pytest.raises(
-            _InvalidOutput,
-            match=r"\.process_spider_output must return an iterable, got <class 'NoneType'>",
-        ):
-            await self._get_middleware_result(ProcessSpiderOutputNonIterableMiddleware)
-
-    @coroutine_test
-    async def test_coroutine(self):
-        with pytest.raises(
-            _InvalidOutput,
-            match=r"\.process_spider_output must be an asynchronous generator",
-        ):
-            await self._get_middleware_result(ProcessSpiderOutputCoroutineMiddleware)
-
-
 class ProcessStartSimpleMiddleware:
     async def process_start(self, start):
         async for item_or_request in start:
@@ -415,11 +320,11 @@ class TestUniversalMiddlewareManager:
         return SpiderMiddlewareManager.from_crawler(crawler)
 
     def test_simple_mw(self, mwman: SpiderMiddlewareManager) -> None:
-        mw = ProcessSpiderOutputSimpleMiddleware()
+        mw = ProcessSpiderOutputSyncMiddleware()
-        mwman._add_middleware(mw)
+        with pytest.raises(
-        assert (
+            TypeError, match=r"doesn't support asynchronous spider output"
-            mwman.methods["process_spider_output"][0] == mw.process_spider_output  # pylint: disable=comparison-with-callable
+        ):
-        )
+            mwman._add_middleware(mw)
 
     def test_async_mw(self, mwman: SpiderMiddlewareManager) -> None:
         mw = ProcessSpiderOutputAsyncGenMiddleware()
@@ -431,9 +336,8 @@ class TestUniversalMiddlewareManager:
     def test_universal_mw(self, mwman: SpiderMiddlewareManager) -> None:
         mw = ProcessSpiderOutputUniversalMiddleware()
         mwman._add_middleware(mw)
-        assert mwman.methods["process_spider_output"][0] == (
+        assert (
-            mw.process_spider_output,
+            mwman.methods["process_spider_output"][0] == mw.process_spider_output_async  # pylint: disable=comparison-with-callable
-            mw.process_spider_output_async,
         )
 
     def test_universal_mw_no_sync(
@@ -441,7 +345,7 @@ class TestUniversalMiddlewareManager:
     ) -> None:
         mwman._add_middleware(UniversalMiddlewareNoSync())
         assert (
-            "UniversalMiddlewareNoSync has process_spider_output_async"
+            "UniversalMiddlewareNoSync has process_spider_output_async()"
             " without process_spider_output" in caplog.text
         )
         assert mwman.methods["process_spider_output"][0] is None
@@ -465,7 +369,7 @@ class TestUniversalMiddlewareManager:
         mwman._add_middleware(UniversalMiddlewareBothAsync())
         assert (
             "UniversalMiddlewareBothAsync.process_spider_output "
-            "is an async generator function while process_spider_output_async exists"
+            "is an async generator function while process_spider_output_async() exists"
             in caplog.text
         )
         assert mwman.methods["process_spider_output"][0] is None
@@ -473,7 +377,6 @@ class TestUniversalMiddlewareManager:
 
 class TestBuiltinMiddlewareSimple(TestBaseAsyncSpiderMiddleware):
     ITEM_TYPE = dict
-    MW_SIMPLE = ProcessSpiderOutputSimpleMiddleware
     MW_ASYNCGEN = ProcessSpiderOutputAsyncGenMiddleware
     MW_UNIVERSAL = ProcessSpiderOutputUniversalMiddleware
 
@@ -488,66 +391,22 @@ class TestBuiltinMiddlewareSimple(TestBaseAsyncSpiderMiddleware):
             self._scrape_func, self.response, self.request
         )
 
-    @coroutine_test
-    async def test_just_builtin(self):
-        await self._test_simple_base()
-
-    @coroutine_test
-    async def test_builtin_simple(self):
-        await self._test_simple_base(self.MW_SIMPLE, start_index=1000)
-
-    @coroutine_test
-    async def test_builtin_async(self):
-        """Upgrade"""
-        await self._test_asyncgen_base(self.MW_ASYNCGEN, start_index=1000)
-
-    @coroutine_test
-    async def test_builtin_universal(self):
-        await self._test_simple_base(self.MW_UNIVERSAL, start_index=1000)
-
-    @coroutine_test
-    async def test_simple_builtin(self):
-        await self._test_simple_base(self.MW_SIMPLE)
-
-    @coroutine_test
-    async def test_async_builtin(self):
-        """Upgrade"""
-        await self._test_asyncgen_base(self.MW_ASYNCGEN)
-
-    @coroutine_test
-    async def test_universal_builtin(self):
-        await self._test_simple_base(self.MW_UNIVERSAL)
-
-
-class TestBuiltinMiddlewareAsyncGen(TestBuiltinMiddlewareSimple):
-    async def _callback(self) -> Any:
-        for item in super()._callback():
-            yield item
-
     @coroutine_test
     async def test_just_builtin(self):
         await self._test_asyncgen_base()
 
-    @coroutine_test
-    async def test_builtin_simple(self):
-        """Downgrade"""
-        await self._test_simple_base(self.MW_SIMPLE, downgrade=True, start_index=1000)
-
     @coroutine_test
     async def test_builtin_async(self):
+        """Upgrade"""
         await self._test_asyncgen_base(self.MW_ASYNCGEN, start_index=1000)
 
     @coroutine_test
     async def test_builtin_universal(self):
         await self._test_asyncgen_base(self.MW_UNIVERSAL, start_index=1000)
 
-    @coroutine_test
-    async def test_simple_builtin(self):
-        """Downgrade"""
-        await self._test_simple_base(self.MW_SIMPLE, downgrade=True)
-
     @coroutine_test
     async def test_async_builtin(self):
+        """Upgrade"""
         await self._test_asyncgen_base(self.MW_ASYNCGEN)
 
     @coroutine_test
@@ -555,9 +414,14 @@ class TestBuiltinMiddlewareAsyncGen(TestBuiltinMiddlewareSimple):
         await self._test_asyncgen_base(self.MW_UNIVERSAL)
 
 
+class TestBuiltinMiddlewareAsyncGen(TestBuiltinMiddlewareSimple):
+    async def _callback(self) -> Any:
+        for item in super()._callback():
+            yield item
+
+
 class TestProcessSpiderException(TestBaseAsyncSpiderMiddleware):
     ITEM_TYPE = dict
-    MW_SIMPLE = ProcessSpiderOutputSimpleMiddleware
     MW_ASYNCGEN = ProcessSpiderOutputAsyncGenMiddleware
     MW_UNIVERSAL = ProcessSpiderOutputUniversalMiddleware
     MW_EXC_SIMPLE = ProcessSpiderExceptionSimpleIterableMiddleware
@@ -576,18 +440,13 @@ class TestProcessSpiderException(TestBaseAsyncSpiderMiddleware):
     @coroutine_test
     async def test_exc_simple(self):
         """Simple exc mw"""
-        await self._test_simple_base(self.MW_EXC_SIMPLE)
+        await self._test_asyncgen_base(self.MW_EXC_SIMPLE)
 
     @coroutine_test
     async def test_exc_async(self):
         """Async exc mw"""
         await self._test_asyncgen_base(self.MW_EXC_ASYNCGEN)
 
-    @coroutine_test
-    async def test_exc_simple_simple(self):
-        """Simple exc mw -> simple output mw"""
-        await self._test_simple_base(self.MW_SIMPLE, self.MW_EXC_SIMPLE)
-
     @coroutine_test
     async def test_exc_async_async(self):
         """Async exc mw -> async output mw"""
@@ -598,25 +457,27 @@ class TestProcessSpiderException(TestBaseAsyncSpiderMiddleware):
         """Simple exc mw -> async output mw; upgrade"""
         await self._test_asyncgen_base(self.MW_ASYNCGEN, self.MW_EXC_SIMPLE)
 
-    @coroutine_test
-    async def test_exc_async_simple(self):
-        """Async exc mw -> simple output mw; cannot work as downgrading is not supported"""
-        await self._test_asyncgen_nodowngrade(self.MW_SIMPLE, self.MW_EXC_ASYNCGEN)
-
 
 class TestDeprecatedSpiderArg(TestSpiderMiddleware):
     @coroutine_test
     async def test_deprecated_mw_spider_arg(self):
-        class DeprecatedSpiderArgMiddleware:
+        class DeprecatedSpiderArgMiddleware1:
             def process_spider_input(self, response, spider):
                 return None
 
-            def process_spider_output(self, response, result, spider):
+            async def process_spider_output(self, response, result, spider):
                 1 / 0
+                yield
 
+        class DeprecatedSpiderArgMiddleware2:
             def process_spider_exception(self, response, exception, spider):
                 return []
 
+        with pytest.warns(
+            ScrapyDeprecationWarning,
+            match=r"process_spider_exception\(\) requires a spider argument",
+        ):
+            self.mwman._add_middleware(DeprecatedSpiderArgMiddleware2())
         with (
             pytest.warns(
                 ScrapyDeprecationWarning,
@@ -626,13 +487,10 @@ class TestDeprecatedSpiderArg(TestSpiderMiddleware):
                 ScrapyDeprecationWarning,
                 match=r"process_spider_output\(\) requires a spider argument",
             ),
-            pytest.warns(
-                ScrapyDeprecationWarning,
-                match=r"process_spider_exception\(\) requires a spider argument",
-            ),
         ):
-            self.mwman._add_middleware(DeprecatedSpiderArgMiddleware())
+            self.mwman._add_middleware(DeprecatedSpiderArgMiddleware1())
-        await self._scrape_response()
+        it = await self._scrape_response()
+        await collect_asyncgen(it)
 
     @coroutine_test
     async def test_deprecated_mwman_spider_arg(self):

tests/test_spidermiddleware_output_chain.py

@@ -169,8 +169,8 @@ class NotGeneratorCallbackSpiderMiddlewareRightAfterSpider(NotGeneratorCallbackS
 # ================================================================================
 # (4) exceptions from a middleware process_spider_output method (generator)
 class _GeneratorDoNothingMiddleware(_BaseSpiderMiddleware):
-    def process_spider_output(self, response, result):
+    async def process_spider_output(self, response, result):
-        for r in result:
+        async for r in result:
             r["processed"].append(f"{self.__class__.__name__}.process_spider_output")
             yield r
 
@@ -182,8 +182,8 @@ class _GeneratorDoNothingMiddleware(_BaseSpiderMiddleware):
 
 
 class GeneratorFailMiddleware(_BaseSpiderMiddleware):
-    def process_spider_output(self, response, result):
+    async def process_spider_output(self, response, result):
-        for r in result:
+        async for r in result:
             r["processed"].append(f"{self.__class__.__name__}.process_spider_output")
             yield r
             raise LookupError
@@ -201,8 +201,8 @@ class GeneratorDoNothingAfterFailureMiddleware(_GeneratorDoNothingMiddleware):
 
 
 class GeneratorRecoverMiddleware(_BaseSpiderMiddleware):
-    def process_spider_output(self, response, result):
+    async def process_spider_output(self, response, result):
-        for r in result:
+        async for r in result:
             r["processed"].append(f"{self.__class__.__name__}.process_spider_output")
             yield r
 
@@ -237,86 +237,6 @@ class GeneratorOutputChainSpider(Spider):
         yield {"processed": ["parse-second-item"]}
 
 
-# ================================================================================
-# (5) exceptions from a middleware process_spider_output method (not generator)
-
-
-class _NotGeneratorDoNothingMiddleware(_BaseSpiderMiddleware):
-    def process_spider_output(self, response, result):
-        out = []
-        for r in result:
-            r["processed"].append(f"{self.__class__.__name__}.process_spider_output")
-            out.append(r)
-        return out
-
-    def process_spider_exception(self, response, exception):
-        method = f"{self.__class__.__name__}.process_spider_exception"
-        self.crawler.spider.logger.info(
-            "%s: %s caught", method, exception.__class__.__name__
-        )
-
-
-class NotGeneratorFailMiddleware(_BaseSpiderMiddleware):
-    def process_spider_output(self, response, result):
-        out = []
-        for r in result:
-            r["processed"].append(f"{self.__class__.__name__}.process_spider_output")
-            out.append(r)
-        raise ReferenceError
-
-    def process_spider_exception(self, response, exception):
-        method = f"{self.__class__.__name__}.process_spider_exception"
-        self.crawler.spider.logger.info(
-            "%s: %s caught", method, exception.__class__.__name__
-        )
-        return [{"processed": [method]}]
-
-
-class NotGeneratorDoNothingAfterFailureMiddleware(_NotGeneratorDoNothingMiddleware):
-    pass
-
-
-class NotGeneratorRecoverMiddleware(_BaseSpiderMiddleware):
-    def process_spider_output(self, response, result):
-        out = []
-        for r in result:
-            r["processed"].append(f"{self.__class__.__name__}.process_spider_output")
-            out.append(r)
-        return out
-
-    def process_spider_exception(self, response, exception):
-        method = f"{self.__class__.__name__}.process_spider_exception"
-        self.crawler.spider.logger.info(
-            "%s: %s caught", method, exception.__class__.__name__
-        )
-        return [{"processed": [method]}]
-
-
-class NotGeneratorDoNothingAfterRecoveryMiddleware(_NotGeneratorDoNothingMiddleware):
-    pass
-
-
-class NotGeneratorOutputChainSpider(Spider):
-    name = "NotGeneratorOutputChainSpider"
-    custom_settings = {
-        "SPIDER_MIDDLEWARES": {
-            NotGeneratorFailMiddleware: 10,
-            NotGeneratorDoNothingAfterFailureMiddleware: 8,
-            NotGeneratorRecoverMiddleware: 5,
-            NotGeneratorDoNothingAfterRecoveryMiddleware: 3,
-        },
-    }
-
-    async def start(self):
-        yield Request(self.mockserver.url("/status?n=200"))
-
-    def parse(self, response):
-        return [
-            {"processed": ["parse-first-item"]},
-            {"processed": ["parse-second-item"]},
-        ]
-
-
 # ================================================================================
 class TestSpiderMiddleware:
     mockserver: MockServer
@@ -481,41 +401,3 @@ class TestSpiderMiddleware:
         assert str(item_from_callback) in str(log4)
         assert str(item_recovered) in str(log4)
         assert "parse-second-item" not in str(log4)
-
-    @coroutine_test
-    async def test_not_a_generator_output_chain(self):
-        """
-        (5) An exception from a middleware's process_spider_output method should be sent
-        to the process_spider_exception method from the next middleware in the chain.
-        The result of the recovery by the process_spider_exception method should be handled
-        by the process_spider_output method from the next middleware.
-        The final item count should be 1 (from the process_spider_exception chain, the items
-        from the spider callback are lost)
-        """
-        log5 = await self.crawl_log(NotGeneratorOutputChainSpider)
-        assert "'item_scraped_count': 1" in str(log5)
-        assert (
-            "GeneratorRecoverMiddleware.process_spider_exception: ReferenceError caught"
-            in str(log5)
-        )
-        assert (
-            "GeneratorDoNothingAfterFailureMiddleware.process_spider_exception: ReferenceError caught"
-            in str(log5)
-        )
-        assert (
-            "GeneratorFailMiddleware.process_spider_exception: ReferenceError caught"
-            not in str(log5)
-        )
-        assert (
-            "GeneratorDoNothingAfterRecoveryMiddleware.process_spider_exception: ReferenceError caught"
-            not in str(log5)
-        )
-        item_recovered = {
-            "processed": [
-                "NotGeneratorRecoverMiddleware.process_spider_exception",
-                "NotGeneratorDoNothingAfterRecoveryMiddleware.process_spider_output",
-            ]
-        }
-        assert str(item_recovered) in str(log5)
-        assert "parse-first-item" not in str(log5)
-        assert "parse-second-item" not in str(log5)

tests/test_utils_python.py

@@ -12,7 +12,6 @@ from scrapy.utils.asyncgen import as_async_generator, collect_asyncgen
 from scrapy.utils.defer import aiter_errback
 from scrapy.utils.python import (
     MutableAsyncChain,
-    MutableChain,
     binary_is_text,
     get_func_args,
     memoizemethod_noargs,
@@ -30,16 +29,6 @@ _KT = TypeVar("_KT")
 _VT = TypeVar("_VT")
 
 
-def test_mutablechain():
-    m = MutableChain(range(2), [2, 3], (4, 5))
-    m.extend(range(6, 7))
-    m.extend([7, 8])
-    m.extend([9, 10], (11, 12))
-    assert next(m) == 0
-    assert m.__next__() == 1
-    assert list(m) == list(range(2, 13))
-
-
 class TestMutableAsyncChain:
     @staticmethod
     async def g1():