3.5 KiB
Coroutines
System Message: ERROR/3 (<stdin>, line 5)
Unknown directive type "versionadded".
.. versionadded:: 2.0
Scrapy has :ref:`partial support <coroutine-support>` for the :ref:`coroutine syntax <async>`.
System Message: ERROR/3 (<stdin>, line 7); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 7); backlink
Unknown interpreted text role "ref".Supported callables
The following callables may be defined as coroutines using async def, and hence use coroutine syntax (e.g. await, async for, async with):
:class:`~scrapy.http.Request` callbacks.
System Message: ERROR/3 (<stdin>, line 18); backlink
Unknown interpreted text role "class".
The :meth:`process_item` method of :ref:`item pipelines <topics-item-pipeline>`.
System Message: ERROR/3 (<stdin>, line 20); backlink
Unknown interpreted text role "meth".
System Message: ERROR/3 (<stdin>, line 20); backlink
Unknown interpreted text role "ref".
The :meth:`~scrapy.downloadermiddlewares.DownloaderMiddleware.process_request`, :meth:`~scrapy.downloadermiddlewares.DownloaderMiddleware.process_response`, and :meth:`~scrapy.downloadermiddlewares.DownloaderMiddleware.process_exception` methods of :ref:`downloader middlewares <topics-downloader-middleware-custom>`.
System Message: ERROR/3 (<stdin>, line 23); backlink
Unknown interpreted text role "meth".
System Message: ERROR/3 (<stdin>, line 23); backlink
Unknown interpreted text role "meth".
System Message: ERROR/3 (<stdin>, line 23); backlink
Unknown interpreted text role "meth".
System Message: ERROR/3 (<stdin>, line 23); backlink
Unknown interpreted text role "ref".
:ref:`Signal handlers that support deferreds <signal-deferred>`.
System Message: ERROR/3 (<stdin>, line 31); backlink
Unknown interpreted text role "ref".
Usage
There are several use cases for coroutines in Scrapy. Code that would return Deferreds when written for previous Scrapy versions, such as downloader middlewares and signal handlers, can be rewritten to be shorter and cleaner:
from itemadapter import ItemAdapter
class DbPipeline:
def _update_item(self, data, item):
adapter = ItemAdapter(item)
adapter['field'] = data
return item
def process_item(self, item, spider):
adapter = ItemAdapter(item)
dfd = db.get_some_data(adapter['id'])
dfd.addCallback(self._update_item, item)
return dfd
becomes:
from itemadapter import ItemAdapter
class DbPipeline:
async def process_item(self, item, spider):
adapter = ItemAdapter(item)
adapter['field'] = await db.get_some_data(adapter['id'])
return item
Coroutines may be used to call asynchronous code. This includes other coroutines, functions that return Deferreds and functions that return :term:`awaitable objects <awaitable>` such as :class:`~asyncio.Future`. This means you can use many useful Python libraries providing such code:
System Message: ERROR/3 (<stdin>, line 64); backlink
Unknown interpreted text role "term".System Message: ERROR/3 (<stdin>, line 64); backlink
Unknown interpreted text role "class".class MySpider(Spider):
# ...
async def parse_with_deferred(self, response):
additional_response = await treq.get('https://additional.url')
additional_data = await treq.content(additional_response)
# ... use response and additional_data to yield items and requests
async def parse_with_asyncio(self, response):
async with aiohttp.ClientSession() as session:
async with session.get('https://additional.url') as additional_response:
additional_data = await r.text()
# ... use response and additional_data to yield items and requests
Note
Many libraries that use coroutines, such as aio-libs, require the :mod:`asyncio` loop and to use them you need to :doc:`enable asyncio support in Scrapy<asyncio>`.
System Message: ERROR/3 (<stdin>, line 82); backlink
Unknown interpreted text role "mod".System Message: ERROR/3 (<stdin>, line 82); backlink
Unknown interpreted text role "doc".Note
If you want to await on Deferreds, you may need to :ref:`wrap them<asyncio-await-dfd>`.
System Message: ERROR/3 (<stdin>, line 86); backlink
Unknown interpreted text role "ref".Common use cases for asynchronous code include:
requesting data from websites, databases and other services (in callbacks, pipelines and middlewares);
storing data in databases (in pipelines and middlewares);
delaying the spider initialization until some external event (in the :signal:`spider_opened` handler);
System Message: ERROR/3 (<stdin>, line 94); backlink
Unknown interpreted text role "signal".
calling asynchronous Scrapy methods like ExecutionEngine.download (see :ref:`the screenshot pipeline example<ScreenshotPipeline>`).
System Message: ERROR/3 (<stdin>, line 96); backlink
Unknown interpreted text role "ref".