RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Remove support for sync process_spider_output(). (#7504) 

* Remove support for sync process_spider_output().

* Slight check fix.

* Don't expect exceptions from calling process_spider_output().

* Remove dead code.

* Fix test_deprecated_mw_spider_arg() to run all methods.

* Fix typing.

* Typos.

* Remove references to the removed section, move the universal section to the spider middleware docs and write it from a different perspective

* Fix indentation issue

* Minor rewordings

* older versions → lower versions

* Update news.rst

---------

Co-authored-by: Adrian Chaves <adrian@zyte.com>
This commit is contained in:
Andrey Rakhmatullin 2026-05-15 16:36:05 +05:00 committed by GitHub
parent 85c616c5c7
commit a8a8f20d9c
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
10 changed files with 170 additions and 720 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

21
docs/news.rst
View File

@ -1380,8 +1380,8 @@ Highlights:
- Added the :reqmeta:`allow_offsite` request meta key
- :ref:`Spider middlewares that don't support asynchronous spider output
<sync-async-spider-middleware>` are deprecated
- Spider middlewares that don't support asynchronous spider output are
deprecated
- Added a base class for :ref:`universal spider middlewares
<universal-spider-middleware>`
@ -1519,13 +1519,11 @@ Deprecations
``start_queue_cls`` parameter.
(:issue:`6752`)
- :ref:`Spider middlewares that don't support asynchronous spider output
<sync-async-spider-middleware>` are deprecated. The async iterable
downgrading feature, needed for using such middlewares with asynchronous
callbacks and with other spider middlewares that produce asynchronous
iterables, is also deprecated. Please update all such middlewares to
support asynchronous spider output.
(:issue:`6664`)
- Spider middlewares that don't support asynchronous spider output are
deprecated. The async iterable downgrading feature, needed for using such
middlewares with asynchronous callbacks and with other spider middlewares
that produce asynchronous iterables, is also deprecated. Please update all
such middlewares to support asynchronous spider output. (:issue:`6664`)
- Functions that were imported from :mod:`w3lib.url` and re-exported in
:mod:`scrapy.utils.url` are now deprecated, you should import them from
@ -1784,9 +1782,8 @@ Documentation
- Documented the setting values set in the default project template.
(:issue:`6762`, :issue:`6775`)
- Improved the :ref:`docs <sync-async-spider-middleware>` about asynchronous
iterable support in spider middlewares.
(:issue:`6688`)
- Improved the docs about asynchronous iterable support in spider
middlewares. (:issue:`6688`)
- Improved the :ref:`docs <coroutine-deferred-apis>` about using
:class:`~twisted.internet.defer.Deferred`-based APIs in coroutine-based

146
docs/topics/coroutines.rst
View File

@ -23,9 +23,6 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):
- :class:`~scrapy.Request` callbacks.
If you are using any custom or third-party :ref:`spider middleware
<topics-spider-middleware>`, see :ref:`sync-async-spider-middleware`.
- The :meth:`process_item` method of
:ref:`item pipelines <topics-item-pipeline>`.
@ -39,13 +36,9 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):
- The
:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output`
method of :ref:`spider middlewares <topics-spider-middleware>`.
If defined as a coroutine, it must be an :term:`asynchronous generator`.
The input ``result`` parameter is an :term:`asynchronous iterable`.
See also :ref:`sync-async-spider-middleware` and
:ref:`universal-spider-middleware`.
method of :ref:`spider middlewares <topics-spider-middleware>`, which
*must* be defined as an :term:`asynchronous generator` except in
:ref:`universal spider middlewares <universal-spider-middleware>`.
- The :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start` method
of :ref:`spider middlewares <custom-spider-middleware>`, which *must* be
@ -277,136 +270,3 @@ You can also send multiple requests in parallel:
"price": responses[0][1].css(".price::text").get(),
"price2": responses[1][1].css(".color::text").get(),
}
.. _sync-async-spider-middleware:
Mixing synchronous and asynchronous spider middlewares
======================================================
The output of a :class:`~scrapy.Request` callback is passed as the ``result``
parameter to the
:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output` method
of the first :ref:`spider middleware <topics-spider-middleware>` from the
:ref:`list of active spider middlewares <topics-spider-middleware-setting>`.
Then the output of that ``process_spider_output`` method is passed to the
``process_spider_output`` method of the next spider middleware, and so on for
every active spider middleware.
Scrapy supports mixing :ref:`coroutine methods <async>` and synchronous methods
in this chain of calls.
However, if any of the ``process_spider_output`` methods is defined as a
synchronous method, and the previous ``Request`` callback or
``process_spider_output`` method is a coroutine, there are some drawbacks to
the asynchronous-to-synchronous conversion that Scrapy does so that the
synchronous ``process_spider_output`` method gets a synchronous iterable as its
``result`` parameter:
- The whole output of the previous ``Request`` callback or
``process_spider_output`` method is awaited at this point.
- If an exception raises while awaiting the output of the previous
``Request`` callback or ``process_spider_output`` method, none of that
output will be processed.
This contrasts with the regular behavior, where all items yielded before
an exception raises are processed.
Asynchronous-to-synchronous conversions are supported for backward
compatibility, but they are deprecated and will stop working in a future
version of Scrapy.
To avoid asynchronous-to-synchronous conversions, when defining ``Request``
callbacks as coroutine methods or when using spider middlewares whose
``process_spider_output`` method is an :term:`asynchronous generator`, all
active spider middlewares must either have their ``process_spider_output``
method defined as an asynchronous generator or :ref:`define a
process_spider_output_async method <universal-spider-middleware>`.
.. _sync-async-spider-middleware-users:
For middleware users
--------------------
If you have asynchronous callbacks or use asynchronous-only spider middlewares
you should make sure the asynchronous-to-synchronous conversions
:ref:`described above <sync-async-spider-middleware>` don't happen. To do this,
make sure all spider middlewares you use support asynchronous spider output.
Even if you don't have asynchronous callbacks and don't use asynchronous-only
spider middlewares in your project, it's still a good idea to make sure all
middlewares you use support asynchronous spider output, so that it will be easy
to start using asynchronous callbacks in the future. Because of this, Scrapy
logs a warning when it detects a synchronous-only spider middleware.
If you want to update middlewares you wrote, see the :ref:`following section
<sync-async-spider-middleware-authors>`. If you have 3rd-party middlewares that
aren't yet updated by their authors, you can :ref:`subclass <tut-inheritance>`
them to make them :ref:`universal <universal-spider-middleware>` and use the
subclasses in your projects.
.. _sync-async-spider-middleware-authors:
For middleware authors
----------------------
If you have a spider middleware that defines a synchronous
``process_spider_output`` method, you should update it to support asynchronous
spider output for :ref:`better compatibility <sync-async-spider-middleware>`,
even if you don't yet use it with asynchronous callbacks, especially if you
publish this middleware for other people to use. You have two options for this:
1. Make the middleware asynchronous, by making the ``process_spider_output``
method an :term:`asynchronous generator`.
2. Make the middleware universal, as described in the :ref:`next section
<universal-spider-middleware>`.
If your middleware won't be used in projects with synchronous-only middlewares,
e.g. because it's an internal middleware and you know that all other
middlewares in your projects are already updated, it's safe to choose the first
option. Otherwise, it's better to choose the second option.
.. _universal-spider-middleware:
Universal spider middlewares
----------------------------
To allow writing a spider middleware that supports asynchronous execution of
its ``process_spider_output`` method in Scrapy 2.7 and later (avoiding
:ref:`asynchronous-to-synchronous conversions <sync-async-spider-middleware>`)
while maintaining support for older Scrapy versions, you may define
``process_spider_output`` as a synchronous method and define an
:term:`asynchronous generator` version of that method with an alternative name:
``process_spider_output_async``.
For example:
.. code-block:: python
class UniversalSpiderMiddleware:
def process_spider_output(self, response, result):
for r in result:
# ... do something with r
yield r
async def process_spider_output_async(self, response, result):
async for r in result:
# ... do something with r
yield r
.. note:: This is an interim measure to allow, for a time, to write code that
works in Scrapy 2.7 and later without requiring
asynchronous-to-synchronous conversions, and works in earlier Scrapy
versions as well.
In some future version of Scrapy, however, this feature will be
deprecated and, eventually, in a later version of Scrapy, this
feature will be removed, and all spider middlewares will be expected
to define their ``process_spider_output`` method as an asynchronous
generator.
Since 2.13.0, Scrapy provides a base class,
:class:`~scrapy.spidermiddlewares.base.BaseSpiderMiddleware`, which implements
the ``process_spider_output()`` and ``process_spider_output_async()`` methods,
so instead of duplicating the processing code you can override the
``get_processed_request()`` and/or the ``get_processed_item()`` method.

63
docs/topics/spider-middleware.rst
View File

@ -117,36 +117,28 @@ one or more of these methods:
:type response: :class:`~scrapy.http.Response` object
.. method:: process_spider_output(response, result)
:async:
This method is called with the results returned from the Spider, after
it has processed the response.
This method is an :term:`asynchronous generator` called with the
results from the spider after the spider has processed the response.
:meth:`process_spider_output` must return an iterable of
:class:`~scrapy.Request` objects and :ref:`item objects
<topics-items>`.
Consider defining this method as an :term:`asynchronous generator`,
which will be a requirement in a future version of Scrapy. However, if
you plan on sharing your spider middleware with other people, consider
either :ref:`enforcing Scrapy 2.7 <enforce-component-requirements>`
as a minimum requirement of your spider middleware, or :ref:`making
your spider middleware universal <universal-spider-middleware>` so that
it works with Scrapy versions earlier than Scrapy 2.7.
.. seealso:: :ref:`universal-spider-middleware`.
:param response: the response which generated this output from the
spider
:type response: :class:`~scrapy.http.Response` object
:param result: the result returned by the spider
:type result: an iterable of :class:`~scrapy.Request` objects and
:ref:`item objects <topics-items>`
:param result: the results from the spider
:type result: an :term:`asynchronous iterable` of
:class:`~scrapy.Request` objects and :ref:`item objects
<topics-items>`
.. method:: process_spider_output_async(response, result)
:async:
If defined, this method must be an :term:`asynchronous generator`,
which will be called instead of :meth:`process_spider_output` if
``result`` is an :term:`asynchronous iterable`.
Alternative name for :meth:`process_spider_output` used when
implementing a :ref:`universal spider middleware
<universal-spider-middleware>`.
.. method:: process_spider_exception(response, exception)
@ -174,13 +166,40 @@ one or more of these methods:
:type exception: :exc:`Exception` object
.. _universal-spider-middleware:
Universal spider middlewares
----------------------------
In Scrapy 2.6.3 and lower, ``process_spider_output()`` must be a *synchronous*
generator.
To support those versions and higher Scrapy versions in the same middleware,
rename your asynchronous :method:`~SpiderMiddleware.process_spider_output()`
method to :method:`~SpiderMiddleware.process_spider_output_async()`, and define
a synchronous ``process_spider_output()`` method to be used by 2.6.3 and lower
versions.
For example:
.. code-block:: python
class UniversalSpiderMiddleware:
async def process_spider_output_async(self, response, result):
async for r in result:
# ... do something with r
yield r
def process_spider_output(self, response, result):
for r in result:
# ... do something with r
yield r
Base class for custom spider middlewares
----------------------------------------
Scrapy provides a base class for custom spider middlewares. It's not required
to use it but it can help with simplifying middleware implementations and
reducing the amount of boilerplate code in :ref:`universal middlewares
<universal-spider-middleware>`.
to use it but it can help with simplifying middleware implementations.
.. module:: scrapy.spidermiddlewares.base

253
scrapy/core/spidermw.py
View File

@ -9,29 +9,28 @@ from __future__ import annotations
import logging
from collections.abc import AsyncIterator, Callable, Coroutine, Iterable
from functools import wraps
from inspect import isasyncgenfunction, iscoroutine
from inspect import isasyncgenfunction
from itertools import islice
from typing import TYPE_CHECKING, Any, TypeAlias, TypeVar, cast
from warnings import warn
from twisted.internet.defer import Deferred, inlineCallbacks
from twisted.python.failure import Failure
from scrapy import Request, Spider
from scrapy.exceptions import ScrapyDeprecationWarning, _InvalidOutput
from scrapy.http import Response
from scrapy.middleware import MiddlewareManager
from scrapy.utils.asyncgen import as_async_generator, collect_asyncgen
from scrapy.utils.asyncgen import as_async_generator
from scrapy.utils.conf import build_component_list
from scrapy.utils.defer import (
_defer_sleep_async,
deferred_from_coro,
maybe_deferred_to_future,
)
from scrapy.utils.python import MutableAsyncChain, MutableChain, global_object_name
from scrapy.utils.python import MutableAsyncChain, global_object_name
if TYPE_CHECKING:
from collections.abc import Generator
from twisted.internet.defer import Deferred
from scrapy.settings import BaseSettings
@ -46,10 +45,6 @@ ScrapeFunc: TypeAlias = Callable[
]
def _isiterable(o: Any) -> bool:
return isinstance(o, (Iterable, AsyncIterator))
class SpiderMiddlewareManager(MiddlewareManager):
component_name = "spider middleware"
@ -67,13 +62,10 @@ class SpiderMiddlewareManager(MiddlewareManager):
if hasattr(mw, "process_start"):
self.methods["process_start"].appendleft(mw.process_start)
process_spider_output = self._get_async_method_pair(mw, "process_spider_output")
process_spider_output = self._get_process_spider_output(mw)
self.methods["process_spider_output"].appendleft(process_spider_output)
if callable(process_spider_output):
if process_spider_output is not None:
self._check_mw_method_spider_arg(process_spider_output)
elif isinstance(process_spider_output, tuple):
for m in process_spider_output:
self._check_mw_method_spider_arg(m)
process_spider_exception = getattr(mw, "process_spider_exception", None)
self.methods["process_spider_exception"].appendleft(process_spider_exception)
@ -105,48 +97,7 @@ class SpiderMiddlewareManager(MiddlewareManager):
return await scrape_func(Failure(), request)
return await scrape_func(response, request)
def _evaluate_iterable(
self,
response: Response,
iterable: Iterable[_T] | AsyncIterator[_T],
exception_processor_index: int,
recover_to: MutableChain[_T] | MutableAsyncChain[_T],
) -> Iterable[_T] | AsyncIterator[_T]:
if isinstance(iterable, AsyncIterator):
return self._process_async(
response,
iterable,
exception_processor_index,
cast("MutableAsyncChain[_T]", recover_to),
)
return self._process_sync(
response,
iterable,
exception_processor_index,
cast("MutableChain[_T]", recover_to),
)
def _process_sync(
self,
response: Response,
iterable: Iterable[_T],
exception_processor_index: int,
recover_to: MutableChain[_T],
) -> Iterable[_T]:
try:
yield from iterable
except Exception as ex:
exception_result = cast(
"Failure | MutableChain[_T]",
self._process_spider_exception(response, ex, exception_processor_index),
)
if isinstance(exception_result, Failure):
raise
assert isinstance(recover_to, MutableChain)
recover_to.extend(exception_result)
async def _process_async(
async def _evaluate_iterable(
self,
response: Response,
iterable: AsyncIterator[_T],
@ -157,13 +108,9 @@ class SpiderMiddlewareManager(MiddlewareManager):
async for r in iterable:
yield r
except Exception as ex:
exception_result = cast(
"Failure | MutableAsyncChain[_T]",
self._process_spider_exception(response, ex, exception_processor_index),
exception_result: MutableAsyncChain[_T] = self._process_spider_exception(
response, ex, exception_processor_index
)
if isinstance(exception_result, Failure):
raise
assert isinstance(recover_to, MutableAsyncChain)
recover_to.extend(exception_result)
def _process_spider_exception(
@ -171,7 +118,7 @@ class SpiderMiddlewareManager(MiddlewareManager):
response: Response,
exception: Exception,
start_index: int = 0,
) -> MutableChain[_T] | MutableAsyncChain[_T]:
) -> MutableAsyncChain[_T]:
# don't handle _InvalidOutput exception
if isinstance(exception, _InvalidOutput):
raise exception
@ -181,28 +128,18 @@ class SpiderMiddlewareManager(MiddlewareManager):
for method_index, method in enumerate(method_list, start=start_index):
if method is None:
continue
method = cast("Callable", method)
if method in self._mw_methods_requiring_spider:
result = method(
response=response, exception=exception, spider=self._spider
)
else:
result = method(response=response, exception=exception)
if _isiterable(result):
if isinstance(result, (Iterable, AsyncIterator)):
# stop exception handling by handing control over to the
# process_spider_output chain if an iterable has been returned
dfd: Deferred[MutableChain[_T] | MutableAsyncChain[_T]] = (
self._process_spider_output(response, result, method_index + 1)
)
# _process_spider_output() returns a Deferred only because of downgrading so this can be
# simplified when downgrading is removed.
if dfd.called:
# the result is available immediately if _process_spider_output didn't do downgrading
return cast("MutableChain[_T] | MutableAsyncChain[_T]", dfd.result)
# we forbid waiting here because otherwise we would need to return a deferred from
# _process_spider_exception too, which complicates the architecture
msg = f"Async iterable returned from {global_object_name(method)} cannot be downgraded"
raise _InvalidOutput(msg)
if isinstance(result, Iterable):
result = as_async_generator(result)
return self._process_spider_output(response, result, method_index + 1)
if result is None:
continue
msg = (
@ -212,124 +149,35 @@ class SpiderMiddlewareManager(MiddlewareManager):
raise _InvalidOutput(msg)
raise exception
# This method cannot be made async def, as _process_spider_exception relies on the Deferred result
# being available immediately which doesn't work when it's a wrapped coroutine.
# It also needs @inlineCallbacks only because of downgrading so it can be removed when downgrading is removed.
@inlineCallbacks
def _process_spider_output( # noqa: PLR0912
def _process_spider_output(
self,
response: Response,
result: Iterable[_T] | AsyncIterator[_T],
result: AsyncIterator[_T],
start_index: int = 0,
) -> Generator[Deferred[Any], Any, MutableChain[_T] | MutableAsyncChain[_T]]:
) -> MutableAsyncChain[_T]:
# items in this iterable do not need to go through the process_spider_output
# chain, they went through it already from the process_spider_exception method
recovered: MutableChain[_T] | MutableAsyncChain[_T]
last_result_is_async = isinstance(result, AsyncIterator)
recovered = MutableAsyncChain() if last_result_is_async else MutableChain()
# There are three cases for the middleware: def foo, async def foo, def foo + async def foo_async.
# 1. def foo. Sync iterables are passed as is, async ones are downgraded.
# 2. async def foo. Sync iterables are upgraded, async ones are passed as is.
# 3. def foo + async def foo_async. Iterables are passed to the respective method.
# Storing methods and method tuples in the same list is weird but we should be able to roll this back
# when we drop this compatibility feature.
recovered: MutableAsyncChain[_T] = MutableAsyncChain()
method_list = islice(self.methods["process_spider_output"], start_index, None)
for method_index, method_pair in enumerate(method_list, start=start_index):
if method_pair is None:
for method_index, method in enumerate(method_list, start=start_index):
if method is None:
continue
need_upgrade = need_downgrade = False
if isinstance(method_pair, tuple):
# This tuple handling is only needed until _async compatibility methods are removed.
method_sync, method_async = method_pair
method = method_async if last_result_is_async else method_sync
if method in self._mw_methods_requiring_spider:
result = method(response=response, result=result, spider=self._spider)
else:
method = method_pair
if not last_result_is_async and isasyncgenfunction(method):
need_upgrade = True
elif last_result_is_async and not isasyncgenfunction(method):
need_downgrade = True
try:
if need_upgrade:
# Iterable -> AsyncIterator
result = as_async_generator(result)
elif need_downgrade:
logger.warning(
f"Async iterable passed to {global_object_name(method)} was"
f" downgraded to a non-async one. This is deprecated and will"
f" stop working in a future version of Scrapy. Please see"
f" https://docs.scrapy.org/en/latest/topics/coroutines.html#for-middleware-users"
f" for more information."
)
assert isinstance(result, AsyncIterator)
# AsyncIterator -> Iterable
result = yield deferred_from_coro(collect_asyncgen(result))
if isinstance(recovered, AsyncIterator):
recovered_collected = yield deferred_from_coro(
collect_asyncgen(recovered)
)
recovered = MutableChain(recovered_collected)
# might fail directly if the output value is not a generator
if method in self._mw_methods_requiring_spider:
result = method(
response=response, result=result, spider=self._spider
)
else:
result = method(response=response, result=result)
except Exception as ex:
exception_result: Failure | MutableChain[_T] | MutableAsyncChain[_T] = (
self._process_spider_exception(response, ex, method_index + 1)
)
if isinstance(exception_result, Failure):
raise
return exception_result
if _isiterable(result):
result = self._evaluate_iterable(
response, result, method_index + 1, recovered
)
else:
if iscoroutine(result):
result.close() # Silence warning about not awaiting
msg = (
f"{global_object_name(method)} must be an asynchronous "
f"generator (i.e. use yield)"
)
else:
msg = (
f"{global_object_name(method)} must return an iterable, got "
f"{type(result)}"
)
raise _InvalidOutput(msg)
last_result_is_async = isinstance(result, AsyncIterator)
if last_result_is_async:
return MutableAsyncChain(result, recovered)
return MutableChain(result, recovered) # type: ignore[arg-type]
result = method(response=response, result=result)
result = self._evaluate_iterable(
response, result, method_index + 1, recovered
)
return MutableAsyncChain(result, recovered)
async def _process_callback_output(
self,
response: Response,
result: Iterable[_T] | AsyncIterator[_T],
) -> MutableChain[_T] | MutableAsyncChain[_T]:
recovered: MutableChain[_T] | MutableAsyncChain[_T]
if isinstance(result, AsyncIterator):
recovered = MutableAsyncChain()
else:
recovered = MutableChain()
self, response: Response, result: AsyncIterator[_T]
) -> MutableAsyncChain[_T]:
recovered: MutableAsyncChain[_T] = MutableAsyncChain()
result = self._evaluate_iterable(response, result, 0, recovered)
result = await maybe_deferred_to_future(
cast(
"Deferred[Iterable[_T] | AsyncIterator[_T]]",
self._process_spider_output(response, result),
)
)
if isinstance(result, AsyncIterator):
return MutableAsyncChain(result, recovered)
if isinstance(recovered, AsyncIterator):
recovered_collected = await collect_asyncgen(recovered)
recovered = MutableChain(recovered_collected)
return MutableChain(result, recovered)
result = self._process_spider_output(response, result)
return MutableAsyncChain(result, recovered)
def scrape_response(
self,
@ -340,7 +188,7 @@ class SpiderMiddlewareManager(MiddlewareManager):
response: Response,
request: Request,
spider: Spider,
) -> Deferred[MutableChain[_T] | MutableAsyncChain[_T]]: # pragma: no cover
) -> Deferred[MutableAsyncChain[_T]]: # pragma: no cover
warn(
"SpiderMiddlewareManager.scrape_response() is deprecated, use scrape_response_async() instead",
ScrapyDeprecationWarning,
@ -363,7 +211,7 @@ class SpiderMiddlewareManager(MiddlewareManager):
scrape_func: ScrapeFunc[_T],
response: Response,
request: Request,
) -> MutableChain[_T] | MutableAsyncChain[_T]:
) -> MutableAsyncChain[_T]:
if not self.crawler:
raise RuntimeError(
"scrape_response_async() called on a SpiderMiddlewareManager"
@ -373,7 +221,8 @@ class SpiderMiddlewareManager(MiddlewareManager):
it: Iterable[_T] | AsyncIterator[_T] = await self._process_spider_input(
scrape_func, response, request
)
return await self._process_callback_output(response, it)
ait = it if isinstance(it, AsyncIterator) else as_async_generator(it)
return await self._process_callback_output(response, ait)
except Exception as ex:
await _defer_sleep_async()
return self._process_spider_exception(response, ex)
@ -399,27 +248,23 @@ class SpiderMiddlewareManager(MiddlewareManager):
# This method is only needed until _async compatibility methods are removed.
@staticmethod
def _get_async_method_pair(
mw: Any, methodname: str
) -> Callable | tuple[Callable, Callable] | None:
normal_method: Callable | None = getattr(mw, methodname, None)
methodname_async = methodname + "_async"
async_method: Callable | None = getattr(mw, methodname_async, None)
def _get_process_spider_output(mw: Any) -> Callable | None:
normal_method: Callable | None = getattr(mw, "process_spider_output", None)
async_method: Callable | None = getattr(mw, "process_spider_output_async", None)
if not async_method:
if normal_method and not isasyncgenfunction(normal_method):
logger.warning(
raise TypeError(
f"Middleware {global_object_name(mw.__class__)} doesn't support"
f" asynchronous spider output, this is deprecated and will stop"
f" working in a future version of Scrapy. The middleware should"
f" be updated to support it. Please see"
f" https://docs.scrapy.org/en/latest/topics/coroutines.html#for-middleware-users"
f" for more information."
f" asynchronous spider output. Its process_spider_output() method"
f" should be an async generator function or it should additionally"
f" define a process_spider_output_async() method."
)
return normal_method
if not normal_method:
logger.error(
f"Middleware {global_object_name(mw.__class__)} has {methodname_async} "
f"without {methodname}, skipping this method."
f"Middleware {global_object_name(mw.__class__)} has"
f" process_spider_output_async() without process_spider_output(),"
f" skipping this method. Please rename it to process_spider_output()."
)
return None
if not isasyncgenfunction(async_method):
@ -431,8 +276,8 @@ class SpiderMiddlewareManager(MiddlewareManager):
if isasyncgenfunction(normal_method):
logger.error(
f"{global_object_name(normal_method)} is an async "
f"generator function while {methodname_async} exists, "
f"skipping both methods."
f"generator function while process_spider_output_async() exists, "
f"skipping both methods. Please remove process_spider_output_async()."
)
return None
return normal_method, async_method
return async_method

5
scrapy/middleware.py
View File

@ -50,10 +50,7 @@ class MiddlewareManager(ABC):
)
self.middlewares: tuple[Any, ...] = middlewares
# Only process_spider_output and process_spider_exception can be None.
# Only process_spider_output can be a tuple, and only until _async compatibility methods are removed.
self.methods: dict[str, deque[Callable | tuple[Callable, Callable] | None]] = (
defaultdict(deque)
)
self.methods: dict[str, deque[Callable | None]] = defaultdict(deque)
self._mw_methods_requiring_spider: set[Callable] = set()
for mw in middlewares:
self._add_middleware(mw)

13
scrapy/utils/python.py
View File

@ -8,12 +8,14 @@ import gc
import inspect
import re
import sys
import warnings
import weakref
from collections.abc import AsyncIterator, Iterable, Mapping
from functools import partial, wraps
from itertools import chain
from typing import TYPE_CHECKING, Any, Concatenate, ParamSpec, TypeVar, overload
from scrapy.exceptions import ScrapyDeprecationWarning
from scrapy.utils.asyncgen import as_async_generator
if TYPE_CHECKING:
@ -294,12 +296,13 @@ else:
gc.collect()
class MutableChain(Iterable[_T]):
"""
Thin wrapper around itertools.chain, allowing to add iterables "in-place"
"""
class MutableChain(Iterable[_T]): # pragma: no cover
def __init__(self, *args: Iterable[_T]):
warnings.warn(
"MutableChain is deprecated and will be removed in a future Scrapy version.",
category=ScrapyDeprecationWarning,
stacklevel=2,
)
self.data: Iterator[_T] = chain.from_iterable(args)
def extend(self, *iterables: Iterable[_T]) -> None:

4
tests/test_request_cb_kwargs.py
View File

@ -38,8 +38,8 @@ class InjectArgumentsSpiderMiddleware:
if request.callback.__name__ == "parse_spider_mw":
request.cb_kwargs["from_process_spider_input"] = True
def process_spider_output(self, response, result):
for element in result:
async def process_spider_output(self, response, result):
async for element in result:
if (
isinstance(element, Request)
and element.callback.__name__ == "parse_spider_mw_2"

244
tests/test_spidermiddleware.py
View File

@ -6,7 +6,6 @@ from typing import TYPE_CHECKING, Any, cast
from unittest import mock
import pytest
from testfixtures import LogCapture
from twisted.internet import defer
from scrapy.core.spidermw import SpiderMiddlewareManager
@ -63,20 +62,6 @@ class TestProcessSpiderInputInvalidOutput(TestSpiderMiddleware):
await self._scrape_response()
class TestProcessSpiderOutputInvalidOutput(TestSpiderMiddleware):
"""Invalid return value for process_spider_output method"""
@coroutine_test
async def test_invalid_process_spider_output(self):
class InvalidProcessSpiderOutputMiddleware:
def process_spider_output(self, response, result):
return 1
self.mwman._add_middleware(InvalidProcessSpiderOutputMiddleware())
with pytest.raises(_InvalidOutput):
await self._scrape_response()
class TestProcessSpiderExceptionInvalidOutput(TestSpiderMiddleware):
"""Invalid return value for process_spider_exception method"""
@ -87,13 +72,15 @@ class TestProcessSpiderExceptionInvalidOutput(TestSpiderMiddleware):
return 1
class RaiseExceptionProcessSpiderOutputMiddleware:
def process_spider_output(self, response, result):
async def process_spider_output(self, response, result):
raise RuntimeError
yield # pylint: disable=unreachable
self.mwman._add_middleware(InvalidProcessSpiderOutputExceptionMiddleware())
self.mwman._add_middleware(RaiseExceptionProcessSpiderOutputMiddleware())
it = await self._scrape_response()
with pytest.raises(_InvalidOutput):
await self._scrape_response()
await collect_asyncgen(it)
class TestProcessSpiderExceptionReRaise(TestSpiderMiddleware):
@ -106,13 +93,15 @@ class TestProcessSpiderExceptionReRaise(TestSpiderMiddleware):
return None
class RaiseExceptionProcessSpiderOutputMiddleware:
def process_spider_output(self, response, result):
async def process_spider_output(self, response, result):
1 / 0
yield
self.mwman._add_middleware(ProcessSpiderExceptionReturnNoneMiddleware())
self.mwman._add_middleware(RaiseExceptionProcessSpiderOutputMiddleware())
it = await self._scrape_response()
with pytest.raises(ZeroDivisionError):
await self._scrape_response()
await collect_asyncgen(it)
class TestBaseAsyncSpiderMiddleware(TestSpiderMiddleware):
@ -155,43 +144,19 @@ class TestBaseAsyncSpiderMiddleware(TestSpiderMiddleware):
self._scrape_func, self.response, self.request
)
async def _test_simple_base(
self,
*mw_classes: type[Any],
downgrade: bool = False,
start_index: int | None = None,
) -> None:
with LogCapture() as log:
result = await self._get_middleware_result(
*mw_classes, start_index=start_index
)
assert isinstance(result, Iterable)
result_list = list(result)
assert len(result_list) == self.RESULT_COUNT
assert isinstance(result_list[0], self.ITEM_TYPE)
assert ("downgraded to a non-async" in str(log)) == downgrade
assert ("doesn't support asynchronous spider output" in str(log)) == (
ProcessSpiderOutputSimpleMiddleware in mw_classes
)
async def _test_asyncgen_base(
self,
*mw_classes: type[Any],
downgrade: bool = False,
start_index: int | None = None,
) -> None:
with LogCapture() as log:
result = await self._get_middleware_result(
*mw_classes, start_index=start_index
)
result = await self._get_middleware_result(*mw_classes, start_index=start_index)
assert isinstance(result, AsyncIterator)
result_list = await collect_asyncgen(result)
assert len(result_list) == self.RESULT_COUNT
assert isinstance(result_list[0], self.ITEM_TYPE)
assert ("downgraded to a non-async" in str(log)) == downgrade
class ProcessSpiderOutputSimpleMiddleware:
class ProcessSpiderOutputSyncMiddleware:
def process_spider_output(self, response, result):
yield from result
@ -232,53 +197,36 @@ class TestProcessSpiderOutputSimple(TestBaseAsyncSpiderMiddleware):
"""process_spider_output tests for simple callbacks"""
ITEM_TYPE = dict
MW_SIMPLE = ProcessSpiderOutputSimpleMiddleware
MW_SYNC = ProcessSpiderOutputSyncMiddleware
MW_ASYNCGEN = ProcessSpiderOutputAsyncGenMiddleware
MW_UNIVERSAL = ProcessSpiderOutputUniversalMiddleware
@coroutine_test
async def test_simple(self):
"""Simple mw"""
await self._test_simple_base(self.MW_SIMPLE)
async def test_sync(self):
"""Unsupported sync mw"""
with pytest.raises(
TypeError, match=r"doesn't support asynchronous spider output"
):
await self._get_middleware_result(self.MW_SYNC)
@coroutine_test
async def test_asyncgen(self):
"""Asyncgen mw; upgrade"""
"""Asyncgen mw"""
await self._test_asyncgen_base(self.MW_ASYNCGEN)
@coroutine_test
async def test_simple_asyncgen(self):
"""Simple mw -> asyncgen mw; upgrade"""
await self._test_asyncgen_base(self.MW_ASYNCGEN, self.MW_SIMPLE)
@coroutine_test
async def test_asyncgen_simple(self):
"""Asyncgen mw -> simple mw; upgrade then downgrade"""
await self._test_simple_base(self.MW_SIMPLE, self.MW_ASYNCGEN, downgrade=True)
@coroutine_test
async def test_universal(self):
"""Universal mw"""
await self._test_simple_base(self.MW_UNIVERSAL)
@coroutine_test
async def test_universal_simple(self):
"""Universal mw -> simple mw"""
await self._test_simple_base(self.MW_SIMPLE, self.MW_UNIVERSAL)
@coroutine_test
async def test_simple_universal(self):
"""Simple mw -> universal mw"""
await self._test_simple_base(self.MW_UNIVERSAL, self.MW_SIMPLE)
await self._test_asyncgen_base(self.MW_UNIVERSAL)
@coroutine_test
async def test_universal_asyncgen(self):
"""Universal mw -> asyncgen mw; upgrade"""
"""Universal mw -> asyncgen mw"""
await self._test_asyncgen_base(self.MW_ASYNCGEN, self.MW_UNIVERSAL)
@coroutine_test
async def test_asyncgen_universal(self):
"""Asyncgen mw -> universal mw; upgrade"""
"""Asyncgen mw -> universal mw"""
await self._test_asyncgen_base(self.MW_UNIVERSAL, self.MW_ASYNCGEN)
@ -289,31 +237,6 @@ class TestProcessSpiderOutputAsyncGen(TestProcessSpiderOutputSimple):
for item in super()._callback():
yield item
@coroutine_test
async def test_simple(self):
"""Simple mw; downgrade"""
await self._test_simple_base(self.MW_SIMPLE, downgrade=True)
@coroutine_test
async def test_simple_asyncgen(self):
"""Simple mw -> asyncgen mw; downgrade then upgrade"""
await self._test_asyncgen_base(self.MW_ASYNCGEN, self.MW_SIMPLE, downgrade=True)
@coroutine_test
async def test_universal(self):
"""Universal mw"""
await self._test_asyncgen_base(self.MW_UNIVERSAL)
@coroutine_test
async def test_universal_simple(self):
"""Universal mw -> simple mw; downgrade"""
await self._test_simple_base(self.MW_SIMPLE, self.MW_UNIVERSAL, downgrade=True)
@coroutine_test
async def test_simple_universal(self):
"""Simple mw -> universal mw; downgrade"""
await self._test_simple_base(self.MW_UNIVERSAL, self.MW_SIMPLE, downgrade=True)
class ProcessSpiderOutputNonIterableMiddleware:
def process_spider_output(self, response, result):
@ -325,24 +248,6 @@ class ProcessSpiderOutputCoroutineMiddleware:
return result
class TestProcessSpiderOutputInvalidResult(TestBaseAsyncSpiderMiddleware):
@coroutine_test
async def test_non_iterable(self):
with pytest.raises(
_InvalidOutput,
match=r"\.process_spider_output must return an iterable, got <class 'NoneType'>",
):
await self._get_middleware_result(ProcessSpiderOutputNonIterableMiddleware)
@coroutine_test
async def test_coroutine(self):
with pytest.raises(
_InvalidOutput,
match=r"\.process_spider_output must be an asynchronous generator",
):
await self._get_middleware_result(ProcessSpiderOutputCoroutineMiddleware)
class ProcessStartSimpleMiddleware:
async def process_start(self, start):
async for item_or_request in start:
@ -415,11 +320,11 @@ class TestUniversalMiddlewareManager:
return SpiderMiddlewareManager.from_crawler(crawler)
def test_simple_mw(self, mwman: SpiderMiddlewareManager) -> None:
mw = ProcessSpiderOutputSimpleMiddleware()
mwman._add_middleware(mw)
assert (
mwman.methods["process_spider_output"][0] == mw.process_spider_output # pylint: disable=comparison-with-callable
)
mw = ProcessSpiderOutputSyncMiddleware()
with pytest.raises(
TypeError, match=r"doesn't support asynchronous spider output"
):
mwman._add_middleware(mw)
def test_async_mw(self, mwman: SpiderMiddlewareManager) -> None:
mw = ProcessSpiderOutputAsyncGenMiddleware()
@ -431,9 +336,8 @@ class TestUniversalMiddlewareManager:
def test_universal_mw(self, mwman: SpiderMiddlewareManager) -> None:
mw = ProcessSpiderOutputUniversalMiddleware()
mwman._add_middleware(mw)
assert mwman.methods["process_spider_output"][0] == (
mw.process_spider_output,
mw.process_spider_output_async,
assert (
mwman.methods["process_spider_output"][0] == mw.process_spider_output_async # pylint: disable=comparison-with-callable
)
def test_universal_mw_no_sync(
@ -441,7 +345,7 @@ class TestUniversalMiddlewareManager:
) -> None:
mwman._add_middleware(UniversalMiddlewareNoSync())
assert (
"UniversalMiddlewareNoSync has process_spider_output_async"
"UniversalMiddlewareNoSync has process_spider_output_async()"
" without process_spider_output" in caplog.text
)
assert mwman.methods["process_spider_output"][0] is None
@ -465,7 +369,7 @@ class TestUniversalMiddlewareManager:
mwman._add_middleware(UniversalMiddlewareBothAsync())
assert (
"UniversalMiddlewareBothAsync.process_spider_output "
"is an async generator function while process_spider_output_async exists"
"is an async generator function while process_spider_output_async() exists"
in caplog.text
)
assert mwman.methods["process_spider_output"][0] is None
@ -473,7 +377,6 @@ class TestUniversalMiddlewareManager:
class TestBuiltinMiddlewareSimple(TestBaseAsyncSpiderMiddleware):
ITEM_TYPE = dict
MW_SIMPLE = ProcessSpiderOutputSimpleMiddleware
MW_ASYNCGEN = ProcessSpiderOutputAsyncGenMiddleware
MW_UNIVERSAL = ProcessSpiderOutputUniversalMiddleware
@ -488,66 +391,22 @@ class TestBuiltinMiddlewareSimple(TestBaseAsyncSpiderMiddleware):
self._scrape_func, self.response, self.request
)
@coroutine_test
async def test_just_builtin(self):
await self._test_simple_base()
@coroutine_test
async def test_builtin_simple(self):
await self._test_simple_base(self.MW_SIMPLE, start_index=1000)
@coroutine_test
async def test_builtin_async(self):
"""Upgrade"""
await self._test_asyncgen_base(self.MW_ASYNCGEN, start_index=1000)
@coroutine_test
async def test_builtin_universal(self):
await self._test_simple_base(self.MW_UNIVERSAL, start_index=1000)
@coroutine_test
async def test_simple_builtin(self):
await self._test_simple_base(self.MW_SIMPLE)
@coroutine_test
async def test_async_builtin(self):
"""Upgrade"""
await self._test_asyncgen_base(self.MW_ASYNCGEN)
@coroutine_test
async def test_universal_builtin(self):
await self._test_simple_base(self.MW_UNIVERSAL)
class TestBuiltinMiddlewareAsyncGen(TestBuiltinMiddlewareSimple):
async def _callback(self) -> Any:
for item in super()._callback():
yield item
@coroutine_test
async def test_just_builtin(self):
await self._test_asyncgen_base()
@coroutine_test
async def test_builtin_simple(self):
"""Downgrade"""
await self._test_simple_base(self.MW_SIMPLE, downgrade=True, start_index=1000)
@coroutine_test
async def test_builtin_async(self):
"""Upgrade"""
await self._test_asyncgen_base(self.MW_ASYNCGEN, start_index=1000)
@coroutine_test
async def test_builtin_universal(self):
await self._test_asyncgen_base(self.MW_UNIVERSAL, start_index=1000)
@coroutine_test
async def test_simple_builtin(self):
"""Downgrade"""
await self._test_simple_base(self.MW_SIMPLE, downgrade=True)
@coroutine_test
async def test_async_builtin(self):
"""Upgrade"""
await self._test_asyncgen_base(self.MW_ASYNCGEN)
@coroutine_test
@ -555,9 +414,14 @@ class TestBuiltinMiddlewareAsyncGen(TestBuiltinMiddlewareSimple):
await self._test_asyncgen_base(self.MW_UNIVERSAL)
class TestBuiltinMiddlewareAsyncGen(TestBuiltinMiddlewareSimple):
async def _callback(self) -> Any:
for item in super()._callback():
yield item
class TestProcessSpiderException(TestBaseAsyncSpiderMiddleware):
ITEM_TYPE = dict
MW_SIMPLE = ProcessSpiderOutputSimpleMiddleware
MW_ASYNCGEN = ProcessSpiderOutputAsyncGenMiddleware
MW_UNIVERSAL = ProcessSpiderOutputUniversalMiddleware
MW_EXC_SIMPLE = ProcessSpiderExceptionSimpleIterableMiddleware
@ -576,18 +440,13 @@ class TestProcessSpiderException(TestBaseAsyncSpiderMiddleware):
@coroutine_test
async def test_exc_simple(self):
"""Simple exc mw"""
await self._test_simple_base(self.MW_EXC_SIMPLE)
await self._test_asyncgen_base(self.MW_EXC_SIMPLE)
@coroutine_test
async def test_exc_async(self):
"""Async exc mw"""
await self._test_asyncgen_base(self.MW_EXC_ASYNCGEN)
@coroutine_test
async def test_exc_simple_simple(self):
"""Simple exc mw -> simple output mw"""
await self._test_simple_base(self.MW_SIMPLE, self.MW_EXC_SIMPLE)
@coroutine_test
async def test_exc_async_async(self):
"""Async exc mw -> async output mw"""
@ -598,25 +457,27 @@ class TestProcessSpiderException(TestBaseAsyncSpiderMiddleware):
"""Simple exc mw -> async output mw; upgrade"""
await self._test_asyncgen_base(self.MW_ASYNCGEN, self.MW_EXC_SIMPLE)
@coroutine_test
async def test_exc_async_simple(self):
"""Async exc mw -> simple output mw; cannot work as downgrading is not supported"""
await self._test_asyncgen_nodowngrade(self.MW_SIMPLE, self.MW_EXC_ASYNCGEN)
class TestDeprecatedSpiderArg(TestSpiderMiddleware):
@coroutine_test
async def test_deprecated_mw_spider_arg(self):
class DeprecatedSpiderArgMiddleware:
class DeprecatedSpiderArgMiddleware1:
def process_spider_input(self, response, spider):
return None
def process_spider_output(self, response, result, spider):
async def process_spider_output(self, response, result, spider):
1 / 0
yield
class DeprecatedSpiderArgMiddleware2:
def process_spider_exception(self, response, exception, spider):
return []
with pytest.warns(
ScrapyDeprecationWarning,
match=r"process_spider_exception\(\) requires a spider argument",
):
self.mwman._add_middleware(DeprecatedSpiderArgMiddleware2())
with (
pytest.warns(
ScrapyDeprecationWarning,
@ -626,13 +487,10 @@ class TestDeprecatedSpiderArg(TestSpiderMiddleware):
ScrapyDeprecationWarning,
match=r"process_spider_output\(\) requires a spider argument",
),
pytest.warns(
ScrapyDeprecationWarning,
match=r"process_spider_exception\(\) requires a spider argument",
),
):
self.mwman._add_middleware(DeprecatedSpiderArgMiddleware())
await self._scrape_response()
self.mwman._add_middleware(DeprecatedSpiderArgMiddleware1())
it = await self._scrape_response()
await collect_asyncgen(it)
@coroutine_test
async def test_deprecated_mwman_spider_arg(self):

130
tests/test_spidermiddleware_output_chain.py
View File

@ -169,8 +169,8 @@ class NotGeneratorCallbackSpiderMiddlewareRightAfterSpider(NotGeneratorCallbackS
# ================================================================================
# (4) exceptions from a middleware process_spider_output method (generator)
class _GeneratorDoNothingMiddleware(_BaseSpiderMiddleware):
def process_spider_output(self, response, result):
for r in result:
async def process_spider_output(self, response, result):
async for r in result:
r["processed"].append(f"{self.__class__.__name__}.process_spider_output")
yield r
@ -182,8 +182,8 @@ class _GeneratorDoNothingMiddleware(_BaseSpiderMiddleware):
class GeneratorFailMiddleware(_BaseSpiderMiddleware):
def process_spider_output(self, response, result):
for r in result:
async def process_spider_output(self, response, result):
async for r in result:
r["processed"].append(f"{self.__class__.__name__}.process_spider_output")
yield r
raise LookupError
@ -201,8 +201,8 @@ class GeneratorDoNothingAfterFailureMiddleware(_GeneratorDoNothingMiddleware):
class GeneratorRecoverMiddleware(_BaseSpiderMiddleware):
def process_spider_output(self, response, result):
for r in result:
async def process_spider_output(self, response, result):
async for r in result:
r["processed"].append(f"{self.__class__.__name__}.process_spider_output")
yield r
@ -237,86 +237,6 @@ class GeneratorOutputChainSpider(Spider):
yield {"processed": ["parse-second-item"]}
# ================================================================================
# (5) exceptions from a middleware process_spider_output method (not generator)
class _NotGeneratorDoNothingMiddleware(_BaseSpiderMiddleware):
def process_spider_output(self, response, result):
out = []
for r in result:
r["processed"].append(f"{self.__class__.__name__}.process_spider_output")
out.append(r)
return out
def process_spider_exception(self, response, exception):
method = f"{self.__class__.__name__}.process_spider_exception"
self.crawler.spider.logger.info(
"%s: %s caught", method, exception.__class__.__name__
)
class NotGeneratorFailMiddleware(_BaseSpiderMiddleware):
def process_spider_output(self, response, result):
out = []
for r in result:
r["processed"].append(f"{self.__class__.__name__}.process_spider_output")
out.append(r)
raise ReferenceError
def process_spider_exception(self, response, exception):
method = f"{self.__class__.__name__}.process_spider_exception"
self.crawler.spider.logger.info(
"%s: %s caught", method, exception.__class__.__name__
)
return [{"processed": [method]}]
class NotGeneratorDoNothingAfterFailureMiddleware(_NotGeneratorDoNothingMiddleware):
pass
class NotGeneratorRecoverMiddleware(_BaseSpiderMiddleware):
def process_spider_output(self, response, result):
out = []
for r in result:
r["processed"].append(f"{self.__class__.__name__}.process_spider_output")
out.append(r)
return out
def process_spider_exception(self, response, exception):
method = f"{self.__class__.__name__}.process_spider_exception"
self.crawler.spider.logger.info(
"%s: %s caught", method, exception.__class__.__name__
)
return [{"processed": [method]}]
class NotGeneratorDoNothingAfterRecoveryMiddleware(_NotGeneratorDoNothingMiddleware):
pass
class NotGeneratorOutputChainSpider(Spider):
name = "NotGeneratorOutputChainSpider"
custom_settings = {
"SPIDER_MIDDLEWARES": {
NotGeneratorFailMiddleware: 10,
NotGeneratorDoNothingAfterFailureMiddleware: 8,
NotGeneratorRecoverMiddleware: 5,
NotGeneratorDoNothingAfterRecoveryMiddleware: 3,
},
}
async def start(self):
yield Request(self.mockserver.url("/status?n=200"))
def parse(self, response):
return [
{"processed": ["parse-first-item"]},
{"processed": ["parse-second-item"]},
]
# ================================================================================
class TestSpiderMiddleware:
mockserver: MockServer
@ -481,41 +401,3 @@ class TestSpiderMiddleware:
assert str(item_from_callback) in str(log4)
assert str(item_recovered) in str(log4)
assert "parse-second-item" not in str(log4)
@coroutine_test
async def test_not_a_generator_output_chain(self):
"""
(5) An exception from a middleware's process_spider_output method should be sent
to the process_spider_exception method from the next middleware in the chain.
The result of the recovery by the process_spider_exception method should be handled
by the process_spider_output method from the next middleware.
The final item count should be 1 (from the process_spider_exception chain, the items
from the spider callback are lost)
"""
log5 = await self.crawl_log(NotGeneratorOutputChainSpider)
assert "'item_scraped_count': 1" in str(log5)
assert (
"GeneratorRecoverMiddleware.process_spider_exception: ReferenceError caught"
in str(log5)
)
assert (
"GeneratorDoNothingAfterFailureMiddleware.process_spider_exception: ReferenceError caught"
in str(log5)
)
assert (
"GeneratorFailMiddleware.process_spider_exception: ReferenceError caught"
not in str(log5)
)
assert (
"GeneratorDoNothingAfterRecoveryMiddleware.process_spider_exception: ReferenceError caught"
not in str(log5)
)
item_recovered = {
"processed": [
"NotGeneratorRecoverMiddleware.process_spider_exception",
"NotGeneratorDoNothingAfterRecoveryMiddleware.process_spider_output",
]
}
assert str(item_recovered) in str(log5)
assert "parse-first-item" not in str(log5)
assert "parse-second-item" not in str(log5)

11
tests/test_utils_python.py
View File

@ -12,7 +12,6 @@ from scrapy.utils.asyncgen import as_async_generator, collect_asyncgen
from scrapy.utils.defer import aiter_errback
from scrapy.utils.python import (
MutableAsyncChain,
MutableChain,
binary_is_text,
get_func_args,
memoizemethod_noargs,
@ -30,16 +29,6 @@ _KT = TypeVar("_KT")
_VT = TypeVar("_VT")
def test_mutablechain():
m = MutableChain(range(2), [2, 3], (4, 5))
m.extend(range(6, 7))
m.extend([7, 8])
m.extend([9, 10], (11, 12))
assert next(m) == 0
assert m.__next__() == 1
assert list(m) == list(range(2, 13))
class TestMutableAsyncChain:
@staticmethod
async def g1():