diff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rst index a65bab3ca..57aa3a62d 100644 --- a/docs/topics/coroutines.rst +++ b/docs/topics/coroutines.rst @@ -238,16 +238,52 @@ active spider middlewares must either have their ``process_spider_output`` method defined as an asynchronous generator or :ref:`define a process_spider_output_async method `. -.. note:: When using third-party spider middlewares that only define a - synchronous ``process_spider_output`` method, consider - :ref:`making them universal ` through - :ref:`subclassing `. +.. _sync-async-spider-middleware-users: +For middleware users +-------------------- + +If you have asynchronous callbacks or use asynchronous-only spider middlewares +you should make sure the asynchronous-to-synchronous conversions +:ref:`described above ` don't happen. To do this, +make sure all spider middlewares you use support asynchronous spider output. +Even if you don't have asynchronous callbacks and don't use asynchronous-only +spider middlewares in your project, it's still a good idea to make sure all +middlewares you use support asynchronous spider output, so that it will be easy +to start using asynchronous callbacks in the future. Because of this, Scrapy +logs a warning when it detects a synchronous-only spider middleware. + +If you want to update middlewares you wrote, see the :ref:`following section +`. If you have 3rd-party middlewares that +aren't yet updated by their authors, you can :ref:`subclass ` +them to make them :ref:`universal ` and use the +subclasses in your projects. + +.. _sync-async-spider-middleware-authors: + +For middleware authors +---------------------- + +If you have a spider middleware that defines a synchronous +``process_spider_output`` method, you should update it to support asynchronous +spider output for :ref:`better compatibility `, +even if you don't yet use it with asynchronous callbacks, especially if you +publish this middleware for other people to use. You have two options for this: + +1. Make the middleware asynchronous, by making the ``process_spider_output`` + method an :term:`asynchronous generator`. +2. Make the middleware universal, as described in the :ref:`next section + `. + +If your middleware won't be used in projects with synchronous-only middlewares, +e.g. because it's an internal middleware and you know that all other +middlewares in your projects are already updated, it's safe to choose the first +option. Otherwise, it's better to choose the second option. .. _universal-spider-middleware: Universal spider middlewares -============================ +---------------------------- .. versionadded:: 2.7 diff --git a/scrapy/core/spidermw.py b/scrapy/core/spidermw.py index 86d11c0e0..85a3b5895 100644 --- a/scrapy/core/spidermw.py +++ b/scrapy/core/spidermw.py @@ -227,7 +227,7 @@ class SpiderMiddlewareManager(MiddlewareManager): f"Async iterable passed to {global_object_name(method)} was" f" downgraded to a non-async one. This is deprecated and will" f" stop working in a future version of Scrapy. Please see" - f" https://docs.scrapy.org/en/latest/topics/coroutines.html#mixing-synchronous-and-asynchronous-spider-middlewares" + f" https://docs.scrapy.org/en/latest/topics/coroutines.html#for-middleware-users" f" for more information." ) assert isinstance(result, AsyncIterable) @@ -343,7 +343,7 @@ class SpiderMiddlewareManager(MiddlewareManager): f" asynchronous spider output, this is deprecated and will stop" f" working in a future version of Scrapy. The middleware should" f" be updated to support it. Please see" - f" https://docs.scrapy.org/en/latest/topics/coroutines.html#mixing-synchronous-and-asynchronous-spider-middlewares" + f" https://docs.scrapy.org/en/latest/topics/coroutines.html#for-middleware-users" f" for more information." ) return normal_method