Update docs.

This commit is contained in:
Andrey Rakhmatullin 2021-02-05 16:20:32 +05:00
parent 2152a2a508
commit 58f8481301
2 changed files with 27 additions and 9 deletions

View File

@ -39,4 +39,28 @@ You can also use custom asyncio event loops with the asyncio reactor. Set the
use it instead of the default asyncio event loop.
.. _asyncio-await-dfd:
Awaiting on Deferreds
=====================
When the asyncio reactor isn't installed, you can await on Deferreds in the
coroutines directly. When it is installed, this is not possible anymore, due to
specifics of the Scrapy coroutine integration (the coroutines are wrapped into
asyncio Futures, not into Deferreds directly), and you need to wrap them into
Futures. Scrapy provides two helpers for this:
.. autofunction:: scrapy.utils.defer.deferred_to_future
.. autofunction:: scrapy.utils.defer.maybe_deferred_to_future
If you want to write universal code that works on any reactors,
you should use ``maybe_deferred_to_future`` on all Deferreds::
from scrapy.utils.defer import maybe_deferred_to_future
class MySpider(Spider):
# ...
async def parse_with_deferred(self, response):
additional_response = await maybe_deferred_to_future(treq.get('https://additional.url'))
additional_data = await maybe_deferred_to_future(treq.content(additional_response))
# ... use response and additional_data to yield items and requests

View File

@ -17,15 +17,6 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):
- :class:`~scrapy.http.Request` callbacks.
.. note:: The callback output is not processed until the whole callback
finishes.
As a side effect, if the callback raises an exception, none of its
output is processed.
This is a known caveat of the current implementation that we aim to
address in a future version of Scrapy.
- The :meth:`process_item` method of
:ref:`item pipelines <topics-item-pipeline>`.
@ -92,6 +83,9 @@ This means you can use many useful Python libraries providing such code::
:mod:`asyncio` loop and to use them you need to
:doc:`enable asyncio support in Scrapy<asyncio>`.
.. note:: If you want to ``await`` on Deferreds, you may need to
:ref:`wrap them<asyncio-await-dfd>`.
Common use cases for asynchronous code include:
* requesting data from websites, databases and other services (in callbacks,