RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/docs/topics/coroutines.rst

16 KiB
Raw Blame History

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> </head>

Coroutines

System Message: ERROR/3 (<stdin>, line 7)

Unknown directive type "versionadded".

.. versionadded:: 2.0

Scrapy :ref:`supports <coroutine-support>` the :ref:`coroutine syntax <async>` (i.e. async def).

System Message: ERROR/3 (<stdin>, line 9); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 9); backlink

Unknown interpreted text role "ref".

Supported callables

The following callables may be defined as coroutines using async def, and hence use coroutine syntax (e.g. await, async for, async with):

Using Deferred-based APIs

In addition to native coroutine APIs Scrapy has some APIs that return a :class:`~twisted.internet.defer.Deferred` object or take a user-supplied function that returns a :class:`~twisted.internet.defer.Deferred` object. These APIs are also asynchronous but don't yet support native async def syntax. In the future we plan to add support for the async def syntax to these APIs or replace them with other APIs where changing the existing ones is possible.

System Message: ERROR/3 (<stdin>, line 72); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 72); backlink

Unknown interpreted text role "class".

The following Scrapy methods return :class:`~twisted.internet.defer.Deferred` objects (this list is not complete as it only includes methods that we think may be useful for user code):

System Message: ERROR/3 (<stdin>, line 80); backlink

Unknown interpreted text role "class".

The following user-supplied methods can return :class:`~twisted.internet.defer.Deferred` objects (the methods that can also return coroutines are listed in :ref:`coroutine-support`):

System Message: ERROR/3 (<stdin>, line 111); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 111); backlink

Unknown interpreted text role "ref".

In most cases you can use these APIs in code that otherwise uses coroutines, by wrapping a :class:`~twisted.internet.defer.Deferred` object into a :class:`~asyncio.Future` object or vice versa. See :ref:`asyncio-await-dfd` for more information about this.

System Message: ERROR/3 (<stdin>, line 154); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 154); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 154); backlink

Unknown interpreted text role "ref".

For example:

General usage

There are several use cases for coroutines in Scrapy.

Code that would return Deferreds when written for previous Scrapy versions, such as downloader middlewares and signal handlers, can be rewritten to be shorter and cleaner:

System Message: WARNING/2 (<stdin>, line 183)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    from itemadapter import ItemAdapter


    class DbPipeline:
        def _update_item(self, data, item):
            adapter = ItemAdapter(item)
            adapter["field"] = data
            return item

        def process_item(self, item):
            adapter = ItemAdapter(item)
            dfd = db.get_some_data(adapter["id"])
            dfd.addCallback(self._update_item, item)
            return dfd

becomes:

System Message: WARNING/2 (<stdin>, line 202)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    from itemadapter import ItemAdapter


    class DbPipeline:
        async def process_item(self, item):
            adapter = ItemAdapter(item)
            adapter["field"] = await db.get_some_data(adapter["id"])
            return item

Coroutines may be used to call asynchronous code. This includes other coroutines, functions that return Deferreds and functions that return :term:`awaitable objects <awaitable>` such as :class:`~asyncio.Future`. This means you can use many useful Python libraries providing such code:

System Message: ERROR/3 (<stdin>, line 213); backlink

Unknown interpreted text role "term".

System Message: ERROR/3 (<stdin>, line 213); backlink

Unknown interpreted text role "class".

System Message: WARNING/2 (<stdin>, line 219)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    class MySpiderDeferred(Spider):
        # ...
        async def parse(self, response):
            additional_response = await treq.get("https://additional.url")
            additional_data = await treq.content(additional_response)
            # ... use response and additional_data to yield items and requests


    class MySpiderAsyncio(Spider):
        # ...
        async def parse(self, response):
            async with aiohttp.ClientSession() as session:
                async with session.get("https://additional.url") as additional_response:
                    additional_data = await additional_response.text()
            # ... use response and additional_data to yield items and requests

Note

Many libraries that use coroutines, such as aio-libs, require the :mod:`asyncio` loop and to use them you need to :doc:`enable asyncio support in Scrapy<asyncio>`.

System Message: ERROR/3 (<stdin>, line 237); backlink

Unknown interpreted text role "mod".

System Message: ERROR/3 (<stdin>, line 237); backlink

Unknown interpreted text role "doc".

Note

If you want to await on Deferreds while using the asyncio reactor, you need to :ref:`wrap them<asyncio-await-dfd>`.

System Message: ERROR/3 (<stdin>, line 241); backlink

Unknown interpreted text role "ref".

Common use cases for asynchronous code include:

Inline requests

The spider below shows how to send a request and await its response all from within a spider callback:

System Message: WARNING/2 (<stdin>, line 266)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    from scrapy import Spider, Request


    class SingleRequestSpider(Spider):
        name = "single"
        start_urls = ["https://example.org/product"]

        async def parse(self, response, **kwargs):
            additional_request = Request("https://example.org/price")
            additional_response = await self.crawler.engine.download_async(
                additional_request
            )
            yield {
                "h1": response.css("h1").get(),
                "price": additional_response.css("#price").get(),
            }

You can also send multiple requests in parallel:

System Message: WARNING/2 (<stdin>, line 287)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    import asyncio

    from scrapy import Spider, Request


    class MultipleRequestsSpider(Spider):
        name = "multiple"
        start_urls = ["https://example.com/product"]

        async def parse(self, response, **kwargs):
            additional_requests = [
                Request("https://example.com/price"),
                Request("https://example.com/color"),
            ]
            tasks = []
            for r in additional_requests:
                task = self.crawler.engine.download_async(r)
                tasks.append(task)
            responses = await asyncio.gather(*tasks)
            yield {
                "h1": response.css("h1::text").get(),
                "price": responses[0][1].css(".price::text").get(),
                "price2": responses[1][1].css(".color::text").get(),
            }

Mixing synchronous and asynchronous spider middlewares

System Message: ERROR/3 (<stdin>, line 320)

Unknown directive type "versionadded".

.. versionadded:: 2.7

The output of a :class:`~scrapy.Request` callback is passed as the result parameter to the :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output` method of the first :ref:`spider middleware <topics-spider-middleware>` from the :ref:`list of active spider middlewares <topics-spider-middleware-setting>`. Then the output of that process_spider_output method is passed to the process_spider_output method of the next spider middleware, and so on for every active spider middleware.

System Message: ERROR/3 (<stdin>, line 322); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 322); backlink

Unknown interpreted text role "meth".

System Message: ERROR/3 (<stdin>, line 322); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 322); backlink

Unknown interpreted text role "ref".

Scrapy supports mixing :ref:`coroutine methods <async>` and synchronous methods in this chain of calls.

System Message: ERROR/3 (<stdin>, line 331); backlink

Unknown interpreted text role "ref".

However, if any of the process_spider_output methods is defined as a synchronous method, and the previous Request callback or process_spider_output method is a coroutine, there are some drawbacks to the asynchronous-to-synchronous conversion that Scrapy does so that the synchronous process_spider_output method gets a synchronous iterable as its result parameter:

  • The whole output of the previous Request callback or process_spider_output method is awaited at this point.

  • If an exception raises while awaiting the output of the previous Request callback or process_spider_output method, none of that output will be processed.

    This contrasts with the regular behavior, where all items yielded before an exception raises are processed.

Asynchronous-to-synchronous conversions are supported for backward compatibility, but they are deprecated and will stop working in a future version of Scrapy.

To avoid asynchronous-to-synchronous conversions, when defining Request callbacks as coroutine methods or when using spider middlewares whose process_spider_output method is an :term:`asynchronous generator`, all active spider middlewares must either have their process_spider_output method defined as an asynchronous generator or :ref:`define a process_spider_output_async method <universal-spider-middleware>`.

System Message: ERROR/3 (<stdin>, line 355); backlink

Unknown interpreted text role "term".

System Message: ERROR/3 (<stdin>, line 355); backlink

Unknown interpreted text role "ref".

For middleware users

If you have asynchronous callbacks or use asynchronous-only spider middlewares you should make sure the asynchronous-to-synchronous conversions :ref:`described above <sync-async-spider-middleware>` don't happen. To do this, make sure all spider middlewares you use support asynchronous spider output. Even if you don't have asynchronous callbacks and don't use asynchronous-only spider middlewares in your project, it's still a good idea to make sure all middlewares you use support asynchronous spider output, so that it will be easy to start using asynchronous callbacks in the future. Because of this, Scrapy logs a warning when it detects a synchronous-only spider middleware.

System Message: ERROR/3 (<stdin>, line 367); backlink

Unknown interpreted text role "ref".

If you want to update middlewares you wrote, see the :ref:`following section <sync-async-spider-middleware-authors>`. If you have 3rd-party middlewares that aren't yet updated by their authors, you can :ref:`subclass <tut-inheritance>` them to make them :ref:`universal <universal-spider-middleware>` and use the subclasses in your projects.

System Message: ERROR/3 (<stdin>, line 377); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 377); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 377); backlink

Unknown interpreted text role "ref".

For middleware authors

If you have a spider middleware that defines a synchronous process_spider_output method, you should update it to support asynchronous spider output for :ref:`better compatibility <sync-async-spider-middleware>`, even if you don't yet use it with asynchronous callbacks, especially if you publish this middleware for other people to use. You have two options for this:

System Message: ERROR/3 (<stdin>, line 388); backlink

Unknown interpreted text role "ref".

  1. Make the middleware asynchronous, by making the process_spider_output method an :term:`asynchronous generator`.

    System Message: ERROR/3 (<stdin>, line 394); backlink

    Unknown interpreted text role "term".

  2. Make the middleware universal, as described in the :ref:`next section <universal-spider-middleware>`.

    System Message: ERROR/3 (<stdin>, line 396); backlink

    Unknown interpreted text role "ref".

If your middleware won't be used in projects with synchronous-only middlewares, e.g. because it's an internal middleware and you know that all other middlewares in your projects are already updated, it's safe to choose the first option. Otherwise, it's better to choose the second option.

Universal spider middlewares

System Message: ERROR/3 (<stdin>, line 409)

Unknown directive type "versionadded".

.. versionadded:: 2.7

To allow writing a spider middleware that supports asynchronous execution of its process_spider_output method in Scrapy 2.7 and later (avoiding :ref:`asynchronous-to-synchronous conversions <sync-async-spider-middleware>`) while maintaining support for older Scrapy versions, you may define process_spider_output as a synchronous method and define an :term:`asynchronous generator` version of that method with an alternative name: process_spider_output_async.

System Message: ERROR/3 (<stdin>, line 411); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 411); backlink

Unknown interpreted text role "term".

For example:

System Message: WARNING/2 (<stdin>, line 421)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    class UniversalSpiderMiddleware:
        def process_spider_output(self, response, result):
            for r in result:
                # ... do something with r
                yield r

        async def process_spider_output_async(self, response, result):
            async for r in result:
                # ... do something with r
                yield r

Note

This is an interim measure to allow, for a time, to write code that works in Scrapy 2.7 and later without requiring asynchronous-to-synchronous conversions, and works in earlier Scrapy versions as well.

In some future version of Scrapy, however, this feature will be deprecated and, eventually, in a later version of Scrapy, this feature will be removed, and all spider middlewares will be expected to define their process_spider_output method as an asynchronous generator.

Since 2.13.0, Scrapy provides a base class, :class:`~scrapy.spidermiddlewares.base.BaseSpiderMiddleware`, which implements the process_spider_output() and process_spider_output_async() methods, so instead of duplicating the processing code you can override the get_processed_request() and/or the get_processed_item() method.

System Message: ERROR/3 (<stdin>, line 445); backlink

Unknown interpreted text role "class".
</html>