mirror of https://github.com/scrapy/scrapy.git
Update docs.
This commit is contained in:
parent
2152a2a508
commit
58f8481301
|
|
@ -39,4 +39,28 @@ You can also use custom asyncio event loops with the asyncio reactor. Set the
|
|||
use it instead of the default asyncio event loop.
|
||||
|
||||
|
||||
.. _asyncio-await-dfd:
|
||||
|
||||
Awaiting on Deferreds
|
||||
=====================
|
||||
|
||||
When the asyncio reactor isn't installed, you can await on Deferreds in the
|
||||
coroutines directly. When it is installed, this is not possible anymore, due to
|
||||
specifics of the Scrapy coroutine integration (the coroutines are wrapped into
|
||||
asyncio Futures, not into Deferreds directly), and you need to wrap them into
|
||||
Futures. Scrapy provides two helpers for this:
|
||||
|
||||
.. autofunction:: scrapy.utils.defer.deferred_to_future
|
||||
.. autofunction:: scrapy.utils.defer.maybe_deferred_to_future
|
||||
|
||||
If you want to write universal code that works on any reactors,
|
||||
you should use ``maybe_deferred_to_future`` on all Deferreds::
|
||||
|
||||
from scrapy.utils.defer import maybe_deferred_to_future
|
||||
|
||||
class MySpider(Spider):
|
||||
# ...
|
||||
async def parse_with_deferred(self, response):
|
||||
additional_response = await maybe_deferred_to_future(treq.get('https://additional.url'))
|
||||
additional_data = await maybe_deferred_to_future(treq.content(additional_response))
|
||||
# ... use response and additional_data to yield items and requests
|
||||
|
|
|
|||
|
|
@ -17,15 +17,6 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):
|
|||
|
||||
- :class:`~scrapy.http.Request` callbacks.
|
||||
|
||||
.. note:: The callback output is not processed until the whole callback
|
||||
finishes.
|
||||
|
||||
As a side effect, if the callback raises an exception, none of its
|
||||
output is processed.
|
||||
|
||||
This is a known caveat of the current implementation that we aim to
|
||||
address in a future version of Scrapy.
|
||||
|
||||
- The :meth:`process_item` method of
|
||||
:ref:`item pipelines <topics-item-pipeline>`.
|
||||
|
||||
|
|
@ -92,6 +83,9 @@ This means you can use many useful Python libraries providing such code::
|
|||
:mod:`asyncio` loop and to use them you need to
|
||||
:doc:`enable asyncio support in Scrapy<asyncio>`.
|
||||
|
||||
.. note:: If you want to ``await`` on Deferreds, you may need to
|
||||
:ref:`wrap them<asyncio-await-dfd>`.
|
||||
|
||||
Common use cases for asynchronous code include:
|
||||
|
||||
* requesting data from websites, databases and other services (in callbacks,
|
||||
|
|
|
|||
Loading…
Reference in New Issue