From 58f848130145649a0191e98b182977278e2b9b6c Mon Sep 17 00:00:00 2001 From: Andrey Rakhmatullin Date: Fri, 5 Feb 2021 16:20:32 +0500 Subject: [PATCH] Update docs. --- docs/topics/asyncio.rst | 24 ++++++++++++++++++++++++ docs/topics/coroutines.rst | 12 +++--------- 2 files changed, 27 insertions(+), 9 deletions(-) diff --git a/docs/topics/asyncio.rst b/docs/topics/asyncio.rst index 91e1cca0d..4addaa178 100644 --- a/docs/topics/asyncio.rst +++ b/docs/topics/asyncio.rst @@ -39,4 +39,28 @@ You can also use custom asyncio event loops with the asyncio reactor. Set the use it instead of the default asyncio event loop. +.. _asyncio-await-dfd: +Awaiting on Deferreds +===================== + +When the asyncio reactor isn't installed, you can await on Deferreds in the +coroutines directly. When it is installed, this is not possible anymore, due to +specifics of the Scrapy coroutine integration (the coroutines are wrapped into +asyncio Futures, not into Deferreds directly), and you need to wrap them into +Futures. Scrapy provides two helpers for this: + +.. autofunction:: scrapy.utils.defer.deferred_to_future +.. autofunction:: scrapy.utils.defer.maybe_deferred_to_future + +If you want to write universal code that works on any reactors, +you should use ``maybe_deferred_to_future`` on all Deferreds:: + + from scrapy.utils.defer import maybe_deferred_to_future + + class MySpider(Spider): + # ... + async def parse_with_deferred(self, response): + additional_response = await maybe_deferred_to_future(treq.get('https://additional.url')) + additional_data = await maybe_deferred_to_future(treq.content(additional_response)) + # ... use response and additional_data to yield items and requests diff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rst index 3b1549bd3..279632653 100644 --- a/docs/topics/coroutines.rst +++ b/docs/topics/coroutines.rst @@ -17,15 +17,6 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``): - :class:`~scrapy.http.Request` callbacks. - .. note:: The callback output is not processed until the whole callback - finishes. - - As a side effect, if the callback raises an exception, none of its - output is processed. - - This is a known caveat of the current implementation that we aim to - address in a future version of Scrapy. - - The :meth:`process_item` method of :ref:`item pipelines `. @@ -92,6 +83,9 @@ This means you can use many useful Python libraries providing such code:: :mod:`asyncio` loop and to use them you need to :doc:`enable asyncio support in Scrapy`. +.. note:: If you want to ``await`` on Deferreds, you may need to + :ref:`wrap them`. + Common use cases for asynchronous code include: * requesting data from websites, databases and other services (in callbacks,