From fd08bb6cd99f16c9ce433583d4698801dc7e0ac3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A1n=20Chaves?= Date: Wed, 16 Mar 2022 14:34:57 +0100 Subject: [PATCH] Refactor the asynchronous process_spider_output documentation --- docs/topics/coroutines.rst | 157 +++++++++++++++++------------- docs/topics/spider-middleware.rst | 22 +++-- 2 files changed, 102 insertions(+), 77 deletions(-) diff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rst index 073b6bd9a..361cd5e60 100644 --- a/docs/topics/coroutines.rst +++ b/docs/topics/coroutines.rst @@ -17,9 +17,12 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``): - :class:`~scrapy.Request` callbacks. + If you are using any custom or third-party :ref:`spider middleware + `, see :ref:`sync-async-spider-middleware`. + .. versionchanged:: VERSION - Output of async callbacks is now processed asynchronously instead of collecting - all of it first. + Output of async callbacks is now processed asynchronously instead of + collecting all of it first. - The :meth:`process_item` method of :ref:`item pipelines `. @@ -34,18 +37,26 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``): - :ref:`Signal handlers that support deferreds `. -- The :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output` - method of :ref:`spider middlewares `. See - :ref:`async-spider-middlewares`. +- The + :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output` + method of :ref:`spider middlewares `. + + It must be defined as an :term:`asynchronous generator`. The input + ``result`` parameter is an :term:`asynchronous iterable`. + + See also :ref:`sync-async-spider-middleware` and + :ref:`universal-spider-middleware`. .. versionadded:: VERSION -Usage -===== +General usage +============= -There are several use cases for coroutines in Scrapy. Code that would -return Deferreds when written for previous Scrapy versions, such as downloader -middlewares and signal handlers, can be rewritten to be shorter and cleaner:: +There are several use cases for coroutines in Scrapy. + +Code that would return Deferreds when written for previous Scrapy versions, +such as downloader middlewares and signal handlers, can be rewritten to be +shorter and cleaner:: from itemadapter import ItemAdapter @@ -110,46 +121,73 @@ Common use cases for asynchronous code include: .. _aio-libs: https://github.com/aio-libs -.. _async-spider-middlewares: -Asynchronous spider middlewares -=============================== +.. _sync-async-spider-middleware: + +Mixing synchronous and asynchronous spider middlewares +====================================================== .. versionadded:: VERSION -.. note:: This currently applies to - :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output`. - In the future it will also apply to - :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start_requests`. -Middleware methods discussed here can take and return async iterables. They can -return the same type of iterable or they can take a normal one and return an -async one. If such method needs to return an async iterable it must be an async -generator, not just a coroutine that returns an iterable. +The output of a :class:`~scrapy.Request` callback is passed as the ``result`` +parameter to the +:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output` method +of the first :ref:`spider middleware ` from the +:ref:`list of active spider middlewares `. +Then the output of that ``process_spider_output`` method is passed to the +``process_spider_output`` method of the next spider middleware, and so on for +every active spider middleware. -As the result of a middleware method is passed to the same method of the next -middleware, it needs to be adapted if the second method expects a different -type. Scrapy will do this transparently: +Scrapy supports mixing :ref:`coroutine methods ` and synchronous methods +in this chain of calls. -* A normal iterable is wrapped into an async one which shouldn't cause any side - effects. -* An async iterable is downgraded to a normal one by waiting until all results - are available and wrapping them in a normal iterable. This is problematic - because it pauses the normal middleware processing for this iterable and - because all results can be skipped if exceptions are raised during - processing. This case emits a warning and will be deprecated and then removed - in a later Scrapy version. -* Async iterables returned from - :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_exception` - won't be downgraded, an exception will be raised if that is needed. +However, if any of the ``process_spider_output`` methods is defined as a +synchronous method, and the previous ``Request`` callback or +``process_spider_output`` method is a coroutine, there are some drawbacks to +the asynchronous-to-synchronous conversion that Scrapy does so that the +synchronous ``process_spider_output`` method gets a synchronous iterable as its +``result`` parameter: -As downgrading is undesirable, here is the proposed way to avoid it. If all -middlewares, including 3rd-party ones, support async iterables as input, no -downgrading will happen. But removing normal iterable support (making the -method a coroutine) from a middleware published as a separate project or used -internally in projects for older Scrapy versions breaks backwards -compatibility. So, as an interim measure (it will be deprecated and then -removed in a later Scrapy version), a middleware can provide both sync and -async methods in the following form:: +- The whole output of the previous ``Request`` callback or + ``process_spider_output`` method is awaited at this point. + +- If an exception raises while awaiting the output of the previous + ``Request`` callback or ``process_spider_output`` method, none of that + output will be processed. + +Asynchronous-to-synchronous conversions are supported for backward +compatibility, but they are deprecated and will stop working in a future +version of Scrapy. + +To avoid asynchronous-to-synchronous conversion, when defining ``Request`` +callbacks as coroutine methods or when using spider middlewares whose +``process_spider_output`` method is an :term:`asynchronous generator`, all +active spider middlewares must either have their ``process_spider_output`` +method defined as an asynchronous generator or :ref:`define a +process_spider_output_async method `. + +.. note:: When using third-party spider middlewares that only define a + synchronous ``process_spider_output`` method, consider + :ref:`making them universal ` through + :ref:`subclassing `. + + +.. _universal-spider-middleware: + +Universal spider middleware +=========================== + +.. versionadded:: VERSION + +To allow writing a spider middleware that supports asynchronous execution of +its ``process_spider_output`` method in Scrapy VERSION and later (avoiding +:ref:`asynchronous-to-synchronous conversions `) +while maintaining support for older Scrapy versions, you may define +``process_spider_output`` as a synchronous method and define an +:term:`asynchronous generator` version of that method with an alternative name: +``process_spider_output_async``. + +For example:: class UniversalSpiderMiddleware: def process_spider_output(self, response, result, spider): @@ -162,28 +200,13 @@ async methods in the following form:: # ... do something with r yield r -In this case normal and async iterables will be passed to the respective -methods without any wrapping or downgrading, and in older versions of Scrapy -the coroutine method will just be ignored. When the backwards compatibility is -no longer needed the non-coroutine method can be dropped and the coroutine one -renamed to the normal name. It may be possible to extract common code from both -methods to reduce code duplication, as in the simplest case the only difference -between them will be ``for`` vs ``async for``. +.. note:: This is an interim measure to allow, for a time, to write code that + works in Scrapy VERSION and later without requiring + asynchronous-to-synchronous conversions, and works in earlier Scrapy + versions as well. -So, to recap: - -* If you don't intend to use async callbacks or middlewares containing async - code in your project, nothing should change for you yet. At some point in the - future some of the 3rd-party middlewares you use may drop backwards - compatibility, which shouldn't lead to immediate problems but may be a sign - to start converting your code to ``async def`` too. -* If you maintain a middleware that can be used with projects you can't control - (e.g. one you published for other people to use, or one that needs to support - some old project that can't be modernized), we recommend adding a - ``process_spider_output_async`` method so that the amount of unnecessary - iterable conversions is reduced but no compatibility is broken. -* If you use async callbacks, try to make sure all middlewares support them. - Note that you can modernize 3rd-party middlewares by subclassing them. -* If you want to write and publish a middleware that requires async code, you - should write in the docs that the minimum support Scrapy version is VERSION - (maybe even check this at the run time, using :attr:`scrapy.__version__`). + In some future version of Scrapy, however, this feature will be + deprecated and, eventually, in a later version of Scrapy, this + feature will be removed, and all spider middlewares will be expected + to define their ``process_spider_output`` method as an asynchronous + generator. diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst index edfc2e4bb..787545ed2 100644 --- a/docs/topics/spider-middleware.rst +++ b/docs/topics/spider-middleware.rst @@ -98,10 +98,6 @@ object gives you access, for example, to the :ref:`settings `. .. method:: process_spider_output(response, result, spider) - .. versionchanged:: VERSION - Since VERSION this can take and return an :term:`python:asynchronous - iterable`. - This method is called with the results returned from the Spider, after it has processed the response. @@ -109,8 +105,15 @@ object gives you access, for example, to the :ref:`settings `. :class:`~scrapy.Request` objects and :ref:`item objects `. - .. note:: When defined as a :ref:`coroutine `, this method needs - to be an async generator, not just return an iterable. + .. versionchanged:: VERSION + This method may be defined as an :term:`asynchronous generator`, in + which case ``result`` is an :term:`asynchronous iterable`. + + Consider defining this method as an :term:`asynchronous generator`, + which will be a requirement in a future version of Scrapy. However, if + you wish your spider middleware to work with Scrapy versions earlier + than Scrapy VERSION, :ref:`make your spider middleware universal + ` instead. :param response: the response which generated this output from the spider @@ -127,10 +130,9 @@ object gives you access, for example, to the :ref:`settings `. .. versionadded:: VERSION - If exists, this methid will be called instead of - :meth:`process_spider_output` when ``result`` is an async iterable. - If this method exists, it must be a coroutine while - :meth:`process_spider_output` must not be a coroutine. + If defined, this method must be an :term:`asynchronous generator`, + which will be called instead of :meth:`process_spider_output` if + ``result`` is an :term:`asynchronous iterable`. .. method:: process_spider_exception(response, exception, spider)