RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/docs/topics/coroutines.rst

10 KiB
Raw Blame History

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> </head>

Coroutines

Scrapy :ref:`supports <coroutine-support>` the :ref:`coroutine syntax <async>` (i.e. async def).

System Message: ERROR/3 (<stdin>, line 7); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 7); backlink

Unknown interpreted text role "ref".

Supported callables

The following callables may be defined as coroutines using async def, and hence use coroutine syntax (e.g. await, async for, async with):

Using Deferred-based APIs

In addition to native coroutine APIs Scrapy has some APIs that return a :class:`~twisted.internet.defer.Deferred` object or take a user-supplied function that returns a :class:`~twisted.internet.defer.Deferred` object. These APIs are also asynchronous but don't yet support native async def syntax. In the future we plan to add support for the async def syntax to these APIs or replace them with other APIs where changing the existing ones isn't possible.

System Message: ERROR/3 (<stdin>, line 63); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 63); backlink

Unknown interpreted text role "class".

These APIs have a coroutine-based implementation and a Deferred-based one:

The following user-supplied methods can return :class:`~twisted.internet.defer.Deferred` objects (the methods that can also return coroutines are listed in :ref:`coroutine-support`):

System Message: ERROR/3 (<stdin>, line 87); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 87); backlink

Unknown interpreted text role "ref".

In most cases you can use these APIs in code that otherwise uses coroutines, by wrapping a :class:`~twisted.internet.defer.Deferred` object into a :class:`~asyncio.Future` object or vice versa. See :ref:`asyncio-await-dfd` for more information about this.

System Message: ERROR/3 (<stdin>, line 124); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 124); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 124); backlink

Unknown interpreted text role "ref".

For example: a custom scheduler needs to define an open() method that can return a :class:`~twisted.internet.defer.Deferred` object. You can write a method that works with Deferreds and returns one directly, or you can write a coroutine and convert it into a function that returns a Deferred with :func:`~scrapy.utils.defer.deferred_f_from_coro_f`.

System Message: ERROR/3 (<stdin>, line 129); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 129); backlink

Unknown interpreted text role "func".

General usage

There are several use cases for coroutines in Scrapy.

Code that would return Deferreds when written for previous Scrapy versions, such as downloader middlewares and signal handlers, can be rewritten to be shorter and cleaner:

System Message: WARNING/2 (<stdin>, line 145)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    from itemadapter import ItemAdapter


    class DbPipeline:
        def _update_item(self, data, item):
            adapter = ItemAdapter(item)
            adapter["field"] = data
            return item

        def process_item(self, item):
            adapter = ItemAdapter(item)
            dfd = db.get_some_data(adapter["id"])
            dfd.addCallback(self._update_item, item)
            return dfd

becomes:

System Message: WARNING/2 (<stdin>, line 164)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    from itemadapter import ItemAdapter


    class DbPipeline:
        async def process_item(self, item):
            adapter = ItemAdapter(item)
            adapter["field"] = await db.get_some_data(adapter["id"])
            return item

Coroutines may be used to call asynchronous code. This includes other coroutines, functions that return Deferreds and functions that return :term:`awaitable objects <awaitable>` such as :class:`~asyncio.Future`. This means you can use many useful Python libraries providing such code:

System Message: ERROR/3 (<stdin>, line 175); backlink

Unknown interpreted text role "term".

System Message: ERROR/3 (<stdin>, line 175); backlink

Unknown interpreted text role "class".

System Message: WARNING/2 (<stdin>, line 181)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    class MySpiderDeferred(Spider):
        # ...
        async def parse(self, response):
            additional_response = await treq.get("https://additional.url")
            additional_data = await treq.content(additional_response)
            # ... use response and additional_data to yield items and requests


    class MySpiderAsyncio(Spider):
        # ...
        async def parse(self, response):
            async with aiohttp.ClientSession() as session:
                async with session.get("https://additional.url") as additional_response:
                    additional_data = await additional_response.text()
            # ... use response and additional_data to yield items and requests

Note

Many libraries that use coroutines, such as aio-libs, require the :mod:`asyncio` loop and to use them you need to :doc:`enable asyncio support in Scrapy<asyncio>`.

System Message: ERROR/3 (<stdin>, line 199); backlink

Unknown interpreted text role "mod".

System Message: ERROR/3 (<stdin>, line 199); backlink

Unknown interpreted text role "doc".

Note

If you want to await on Deferreds while using the asyncio reactor, you need to :ref:`wrap them<asyncio-await-dfd>`.

System Message: ERROR/3 (<stdin>, line 203); backlink

Unknown interpreted text role "ref".

Sometimes you need to call code that is not async-friendly at all, such as a third-party client library with no asynchronous support. Calling such code directly from a coroutine would block the underlying event loop, preventing Scrapy from doing any other work while it runs. To avoid this, you can wrap the blocking call with :func:`~scrapy.utils.decorators.inthread`, which runs it in a separate thread and returns an awaitable with the result:

System Message: ERROR/3 (<stdin>, line 206); backlink

Unknown interpreted text role "func".

System Message: WARNING/2 (<stdin>, line 213)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    from scrapy.utils.decorators import inthread


    @inthread
    def blocking_call(value):
        # e.g. a call into a library with no async support
        return some_sync_client.get(value)


    class MySpider(Spider):
        async def parse(self, response):
            result = await blocking_call(response.url)
            # ... use result to yield items and requests

:func:`~scrapy.utils.decorators.inthread` uses :func:`asyncio.to_thread` when asyncio support is enabled, and Twisted's thread pool otherwise, so it works regardless of which reactor your project uses.

System Message: ERROR/3 (<stdin>, line 229); backlink

Unknown interpreted text role "func".

System Message: ERROR/3 (<stdin>, line 229); backlink

Unknown interpreted text role "func".

Common use cases for asynchronous code include:

Inline requests

The spider below shows how to send a request and await its response all from within a spider callback:

System Message: WARNING/2 (<stdin>, line 257)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    from scrapy import Spider, Request


    class SingleRequestSpider(Spider):
        name = "single"
        start_urls = ["https://example.org/product"]

        async def parse(self, response, **kwargs):
            additional_request = Request("https://example.org/price")
            additional_response = await self.crawler.engine.download_async(
                additional_request
            )
            yield {
                "h1": response.css("h1").get(),
                "price": additional_response.css("#price").get(),
            }

You can also send multiple requests in parallel:

System Message: WARNING/2 (<stdin>, line 278)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    import asyncio

    from scrapy import Spider, Request


    class MultipleRequestsSpider(Spider):
        name = "multiple"
        start_urls = ["https://example.com/product"]

        async def parse(self, response, **kwargs):
            additional_requests = [
                Request("https://example.com/price"),
                Request("https://example.com/color"),
            ]
            tasks = []
            for r in additional_requests:
                task = self.crawler.engine.download_async(r)
                tasks.append(task)
            responses = await asyncio.gather(*tasks)
            yield {
                "h1": response.css("h1::text").get(),
                "price": responses[0].css(".price::text").get(),
                "color": responses[1].css(".color::text").get(),
            }
</html>