RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/docs/topics/coroutines.rst

6.9 KiB
Raw Blame History

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> </head>

Coroutines

System Message: ERROR/3 (<stdin>, line 5)

Unknown directive type "versionadded".

.. versionadded:: 2.0

Scrapy has :ref:`partial support <coroutine-support>` for the :ref:`coroutine syntax <async>`.

System Message: ERROR/3 (<stdin>, line 7); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 7); backlink

Unknown interpreted text role "ref".

Supported callables

The following callables may be defined as coroutines using async def, and hence use coroutine syntax (e.g. await, async for, async with):

Usage

There are several use cases for coroutines in Scrapy. Code that would return Deferreds when written for previous Scrapy versions, such as downloader middlewares and signal handlers, can be rewritten to be shorter and cleaner:

from itemadapter import ItemAdapter

class DbPipeline:
    def _update_item(self, data, item):
        adapter = ItemAdapter(item)
        adapter['field'] = data
        return item

    def process_item(self, item, spider):
        adapter = ItemAdapter(item)
        dfd = db.get_some_data(adapter['id'])
        dfd.addCallback(self._update_item, item)
        return dfd

becomes:

from itemadapter import ItemAdapter

class DbPipeline:
    async def process_item(self, item, spider):
        adapter = ItemAdapter(item)
        adapter['field'] = await db.get_some_data(adapter['id'])
        return item

Coroutines may be used to call asynchronous code. This includes other coroutines, functions that return Deferreds and functions that return :term:`awaitable objects <awaitable>` such as :class:`~asyncio.Future`. This means you can use many useful Python libraries providing such code:

System Message: ERROR/3 (<stdin>, line 75); backlink

Unknown interpreted text role "term".

System Message: ERROR/3 (<stdin>, line 75); backlink

Unknown interpreted text role "class".
class MySpiderDeferred(Spider):
    # ...
    async def parse(self, response):
        additional_response = await treq.get('https://additional.url')
        additional_data = await treq.content(additional_response)
        # ... use response and additional_data to yield items and requests

class MySpiderAsyncio(Spider):
    # ...
    async def parse(self, response):
        async with aiohttp.ClientSession() as session:
            async with session.get('https://additional.url') as additional_response:
                additional_data = await additional_response.text()
        # ... use response and additional_data to yield items and requests

Note

Many libraries that use coroutines, such as aio-libs, require the :mod:`asyncio` loop and to use them you need to :doc:`enable asyncio support in Scrapy<asyncio>`.

System Message: ERROR/3 (<stdin>, line 95); backlink

Unknown interpreted text role "mod".

System Message: ERROR/3 (<stdin>, line 95); backlink

Unknown interpreted text role "doc".

Note

If you want to await on Deferreds, you may need to :ref:`wrap them<asyncio-await-dfd>`.

System Message: ERROR/3 (<stdin>, line 99); backlink

Unknown interpreted text role "ref".

Common use cases for asynchronous code include:

  • requesting data from websites, databases and other services (in callbacks, pipelines and middlewares);

  • storing data in databases (in pipelines and middlewares);

  • delaying the spider initialization until some external event (in the :signal:`spider_opened` handler);

    System Message: ERROR/3 (<stdin>, line 107); backlink

    Unknown interpreted text role "signal".

  • calling asynchronous Scrapy methods like ExecutionEngine.download (see :ref:`the screenshot pipeline example<ScreenshotPipeline>`).

    System Message: ERROR/3 (<stdin>, line 109); backlink

    Unknown interpreted text role "ref".

Asynchronous spider middlewares

System Message: ERROR/3 (<stdin>, line 119)

Unknown directive type "versionadded".

.. versionadded:: VERSION

Note

This currently applies to :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output`.

System Message: ERROR/3 (<stdin>, line 120); backlink

Unknown interpreted text role "meth".

Middleware methods discussed here can take and return async iterables. They can return the same type of iterable or they can take a normal one and return an async one. If such method needs to return an async iterable it must be an async generator, not just a coroutine that returns an iterable.

System Message: ERROR/3 (<stdin>, line 128)

Unknown directive type "autofunction".

.. autofunction:: scrapy.utils.asyncgen.as_async_generator

In the simplest form that supports both sync and async input it can be written like this:

from scrapy.utils.asyncgen import as_async_generator

class ProcessSpiderOutputAsyncGenMiddleware:
    async def process_spider_output(self, response, result, spider):
        async for r in as_async_generator(result):
            # ... do something with r
            yield r

If the middleware input (the callback result for process_spider_output) is an async iterable, all middlewares that process it must support it. The built-in ones do, but the ones in your project and 3rd-party ones will need to be updated to support it, as the code that expects a normal iterable will break on an async one. If these middlewares receive an async iterable, they must return one as well. On the other hand, if they receive a normal iterable, they shouldn't break and ideally should return a normal iterable too. There can be several possible implementations of this.

The simplest one, always converting normal iterables to async ones, is provided above. Because a result of a middleware method is passed to the same method of the next middleware, it's only possible to mix middlewares with synchronous and asynchronous implementations of the same method if all synchronous ones are called first (which isn't always possible).

Another option is to make separate methods for normal and async iterables and choose one at run time:

from inspect import isasyncgen

class ProcessSpiderOutputAsyncGenMiddleware:
    def _normal_process_spider_output(self, response, result, spider):
        # ... do something with normal result

    async def _async_process_spider_output(self, response, result, spider):
        # ... do the same with async result

    def process_spider_output(self, response, result, spider):
        if isasyncgen(result):
            return self._async_process_spider_output(self, response, result, spider)
        else:
            return self._normal_process_spider_output(self, response, result, spider)

If you are writing a middleware that you intend to publish or to use in many projects, this is likely the best way to implement it. It may be possible to extract common code from both methods to reduce code duplication, as in the simplest case the only difference between them will be for vs async for.

</html>