RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/docs/topics/coroutines.rst

6.9 KiB
Raw Blame History

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> </head>

Coroutines

System Message: ERROR/3 (<stdin>, line 5)

Unknown directive type "versionadded".

.. versionadded:: 2.0

Scrapy has :ref:`partial support <coroutine-support>` for the :ref:`coroutine syntax <async>`.

System Message: ERROR/3 (<stdin>, line 7); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 7); backlink

Unknown interpreted text role "ref".

Supported callables

The following callables may be defined as coroutines using async def, and hence use coroutine syntax (e.g. await, async for, async with):

Asynchronous start_requests and spider middlewares

System Message: ERROR/3 (<stdin>, line 61)

Unknown directive type "versionadded".

.. versionadded:: 2.2

The :meth:`~scrapy.spiders.Spider.start_requests` spider method can be an asynchronous generator:

System Message: ERROR/3 (<stdin>, line 63); backlink

Unknown interpreted text role "meth".
async def start_requests():
    # ...
    yield scrapy.Request(...)
    # ...

In this case all spider middlewares used with this spider that have the :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start_requests` method must support this: if they receive an asynchronous iterable, they must return one as well. On the other hand, if they receive a normal iterable, they shouldn't break and ideally should return a normal iterable too. There can be several possible implementations of this.

System Message: ERROR/3 (<stdin>, line 71); backlink

Unknown interpreted text role "meth".

First, such universal :meth:`process_start_requests` can be an asynchronous generator itself, and so it will always convert a normal iterable to an asynchronous one. Because a result of a middleware method is passed to the same method of the next middleware, it's only possible to mix middlewares with synchronous and asynchronous :meth:`process_start_requests` if all synchronous ones are called first.

System Message: ERROR/3 (<stdin>, line 78); backlink

Unknown interpreted text role "meth".

System Message: ERROR/3 (<stdin>, line 78); backlink

Unknown interpreted text role "meth".

System Message: ERROR/3 (<stdin>, line 85)

Unknown directive type "autofunction".

.. autofunction:: scrapy.utils.asyncgen.as_async_generator

Here is an example of a universal middleware using this approach:

from scrapy.utils.asyncgen import as_async_generator

class ProcessStartRequestsAsyncGenMiddleware:
    async def process_start_requests(self, start_requests, spider):
        async for req in as_async_generator(start_requests):
            # ... do something with req
            yield req

If this method includes asynchronous code, that code will work even with synchronous :meth:`~scrapy.spiders.Spider.start_requests`.

System Message: ERROR/3 (<stdin>, line 97); backlink

Unknown interpreted text role "meth".

Another option is to make separate methods for normal and asynchronous iterables and choose one at run time:

from inspect import isasyncgen

class ProcessStartRequestsAsyncGenMiddleware:
    def _normal_process_start_requests(self, start_requests, spider):
        # ... do something with normal start_requests

    async def _async_process_start_requests(self, start_requests, spider):
        # ... do something with async start_requests

    def process_start_requests(self, start_requests, spider):
        if isasyncgen(start_requests):
            return self._async_process_start_requests(start_requests, spider)
        else:
            return self._normal_process_start_requests(start_requests, spider)

Usage

There are several use cases for coroutines in Scrapy. Code that would return Deferreds when written for previous Scrapy versions, such as downloader middlewares and signal handlers, can be rewritten to be shorter and cleaner:

from itemadapter import ItemAdapter

class DbPipeline:
    def _update_item(self, data, item):
        adapter = ItemAdapter(item)
        adapter['field'] = data
        return item

    def process_item(self, item, spider):
        adapter = ItemAdapter(item)
        dfd = db.get_some_data(adapter['id'])
        dfd.addCallback(self._update_item, item)
        return dfd

becomes:

from itemadapter import ItemAdapter

class DbPipeline:
    async def process_item(self, item, spider):
        adapter = ItemAdapter(item)
        adapter['field'] = await db.get_some_data(adapter['id'])
        return item

Coroutines may be used to call asynchronous code. This includes other coroutines, functions that return Deferreds and functions that return :term:`awaitable objects <awaitable>` such as :class:`~asyncio.Future`. This means you can use many useful Python libraries providing such code:

System Message: ERROR/3 (<stdin>, line 150); backlink

Unknown interpreted text role "term".

System Message: ERROR/3 (<stdin>, line 150); backlink

Unknown interpreted text role "class".
class MySpider(Spider):
    # ...
    async def parse_with_deferred(self, response):
        additional_response = await treq.get('https://additional.url')
        additional_data = await treq.content(additional_response)
        # ... use response and additional_data to yield items and requests

    async def parse_with_asyncio(self, response):
        async with aiohttp.ClientSession() as session:
            async with session.get('https://additional.url') as additional_response:
                additional_data = await additional_response.text()
        # ... use response and additional_data to yield items and requests

Note

Many libraries that use coroutines, such as aio-libs, require the :mod:`asyncio` loop and to use them you need to :doc:`enable asyncio support in Scrapy<asyncio>`.

System Message: ERROR/3 (<stdin>, line 168); backlink

Unknown interpreted text role "mod".

System Message: ERROR/3 (<stdin>, line 168); backlink

Unknown interpreted text role "doc".

Common use cases for asynchronous code include:

</html>