RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/docs/topics/asyncio.rst

14 KiB
Raw Blame History

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> </head>

asyncio

Scrapy has partial support for :mod:`asyncio`. After you :ref:`install the asyncio reactor <install-asyncio>`, you may use :mod:`asyncio` and :mod:`asyncio`-powered libraries in any :doc:`coroutine <coroutines>`.

System Message: ERROR/3 (<stdin>, line 7); backlink

Unknown interpreted text role "mod".

System Message: ERROR/3 (<stdin>, line 7); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 7); backlink

Unknown interpreted text role "mod".

System Message: ERROR/3 (<stdin>, line 7); backlink

Unknown interpreted text role "mod".

System Message: ERROR/3 (<stdin>, line 7); backlink

Unknown interpreted text role "doc".

Installing the asyncio reactor

To enable :mod:`asyncio` support, your :setting:`TWISTED_REACTOR` setting needs to be set to 'twisted.internet.asyncioreactor.AsyncioSelectorReactor', which is the default value.

System Message: ERROR/3 (<stdin>, line 17); backlink

Unknown interpreted text role "mod".

System Message: ERROR/3 (<stdin>, line 17); backlink

Unknown interpreted text role "setting".

If you are using :class:`~scrapy.crawler.AsyncCrawlerRunner` or :class:`~scrapy.crawler.CrawlerRunner`, you also need to install the :class:`~twisted.internet.asyncioreactor.AsyncioSelectorReactor` reactor manually. You can do that using :func:`~scrapy.utils.reactor.install_reactor`:

System Message: ERROR/3 (<stdin>, line 21); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 21); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 21); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 21); backlink

Unknown interpreted text role "func".

System Message: WARNING/2 (<stdin>, line 28)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    install_reactor("twisted.internet.asyncioreactor.AsyncioSelectorReactor")

Handling a pre-installed reactor

twisted.internet.reactor and some other Twisted imports install the default Twisted reactor as a side effect. Once a Twisted reactor is installed, it is not possible to switch to a different reactor at run time.

If you :ref:`configure the asyncio Twisted reactor <install-asyncio>` and, at run time, Scrapy complains that a different reactor is already installed, chances are you have some such imports in your code.

System Message: ERROR/3 (<stdin>, line 42); backlink

Unknown interpreted text role "ref".

You can usually fix the issue by moving those offending module-level Twisted imports to the method or function definitions where they are used. For example, if you have something like:

System Message: WARNING/2 (<stdin>, line 51)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    from twisted.internet import reactor


    def my_function():
        reactor.callLater(...)

Switch to something like:

System Message: WARNING/2 (<stdin>, line 61)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    def my_function():
        from twisted.internet import reactor

        reactor.callLater(...)

Alternatively, you can try to :ref:`manually install the asyncio reactor <install-asyncio>`, with :func:`~scrapy.utils.reactor.install_reactor`, before those imports happen.

System Message: ERROR/3 (<stdin>, line 68); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 68); backlink

Unknown interpreted text role "func".

Integrating Deferred code and asyncio code

Coroutine functions can await on Deferreds by wrapping them into :class:`asyncio.Future` objects. Scrapy provides two helpers for this:

System Message: ERROR/3 (<stdin>, line 78); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 81)

Unknown directive type "autofunction".

.. autofunction:: scrapy.utils.defer.deferred_to_future

System Message: ERROR/3 (<stdin>, line 82)

Unknown directive type "autofunction".

.. autofunction:: scrapy.utils.defer.maybe_deferred_to_future

Tip

If you don't need to support reactors other than the default :class:`~twisted.internet.asyncioreactor.AsyncioSelectorReactor`, you can use :func:`~scrapy.utils.defer.deferred_to_future`, otherwise you should use :func:`~scrapy.utils.defer.maybe_deferred_to_future`.

System Message: ERROR/3 (<stdin>, line 84); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 84); backlink

Unknown interpreted text role "func".

System Message: ERROR/3 (<stdin>, line 84); backlink

Unknown interpreted text role "func".

Tip

If you need to use these functions in code that aims to be compatible with lower versions of Scrapy that do not provide these functions, down to Scrapy 2.0 (earlier versions do not support :mod:`asyncio`), you can copy the implementation of these functions into your own code.

System Message: ERROR/3 (<stdin>, line 89); backlink

Unknown interpreted text role "mod".

Coroutines and futures can be wrapped into Deferreds (for example, when a Scrapy API requires passing a Deferred to it) using the following helpers:

System Message: ERROR/3 (<stdin>, line 98)

Unknown directive type "autofunction".

.. autofunction:: scrapy.utils.defer.deferred_from_coro

System Message: ERROR/3 (<stdin>, line 99)

Unknown directive type "autofunction".

.. autofunction:: scrapy.utils.defer.deferred_f_from_coro_f

System Message: ERROR/3 (<stdin>, line 100)

Unknown directive type "autofunction".

.. autofunction:: scrapy.utils.defer.ensure_awaitable

Enforcing asyncio as a requirement

If you are writing a :ref:`component <topics-components>` that requires asyncio to work, use :func:`scrapy.utils.asyncio.is_asyncio_available` to :ref:`enforce it as a requirement <enforce-component-requirements>`. For example:

System Message: ERROR/3 (<stdin>, line 108); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 108); backlink

Unknown interpreted text role "func".

System Message: ERROR/3 (<stdin>, line 108); backlink

Unknown interpreted text role "ref".

System Message: WARNING/2 (<stdin>, line 113)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    from scrapy.utils.asyncio import is_asyncio_available


    class MyComponent:
        def __init__(self):
            if not is_asyncio_available():
                raise ValueError(
                    f"{MyComponent.__qualname__} requires the asyncio support. "
                    f"Make sure you have configured the asyncio reactor in the "
                    f"TWISTED_REACTOR setting. See the asyncio documentation "
                    f"of Scrapy for more information."
                )

System Message: ERROR/3 (<stdin>, line 128)

Unknown directive type "autofunction".

.. autofunction:: scrapy.utils.asyncio.is_asyncio_available

System Message: ERROR/3 (<stdin>, line 129)

Unknown directive type "autofunction".

.. autofunction:: scrapy.utils.reactor.is_asyncio_reactor_installed

Using Scrapy without a Twisted reactor

System Message: ERROR/3 (<stdin>, line 137)

Unknown directive type "versionadded".

.. versionadded:: 2.15.0

Warning

This is currently experimental and may not be suitable for production use.

It's possible to use Scrapy without installing a Twisted reactor at all, by setting the :setting:`TWISTED_REACTOR_ENABLED` setting to False. In this mode Scrapy will use the asyncio event loop directly, and most of the Scrapy functionality will work in the same way.

System Message: ERROR/3 (<stdin>, line 142); backlink

Unknown interpreted text role "setting".

Doing this provides several benefits in certain use cases:

Limitations

As some Scrapy features and components require a reactor, they don't work and are disabled without it. Replacements that don't require a reactor may be added in future Scrapy versions. The following features are not available:

Note that importing Twisted modules and, among other things, creating and using :class:`~twisted.internet.defer.Deferred` objects doesn't require a reactor, so code that uses :class:`~twisted.internet.defer.Deferred`, :class:`~twisted.python.failure.Failure` and some other Twisted APIs will not necessarily stop working.

System Message: ERROR/3 (<stdin>, line 190); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 190); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 190); backlink

Unknown interpreted text role "class".

Other differences

When :setting:`TWISTED_REACTOR_ENABLED` is set to False, Scrapy will change the defaults of some other settings:

System Message: ERROR/3 (<stdin>, line 199); backlink

Unknown interpreted text role "setting".

Thus, :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler` is used by default for making HTTP(S) requests. Please refer to its documentation for its differences and limitations compared to :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler`.

System Message: ERROR/3 (<stdin>, line 207); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 207); backlink

Unknown interpreted text role "class".

Additionally, :class:`~scrapy.crawler.AsyncCrawlerProcess` will install a :term:`meta path finder` that prevents :mod:`twisted.internet.reactor` from being imported.

System Message: ERROR/3 (<stdin>, line 212); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 212); backlink

Unknown interpreted text role "term".

System Message: ERROR/3 (<stdin>, line 212); backlink

Unknown interpreted text role "mod".

Adding support to existing code

Code that doesn't directly use Twisted APIs or APIs that depend on Twisted ones doesn't need special support for running without a reactor.

Here are some examples of APIs and patterns that need a replacement:

Scrapy provides unified helpers for some of these examples:

System Message: ERROR/3 (<stdin>, line 253)

Unknown directive type "autofunction".

.. autofunction:: scrapy.utils.asyncio.call_later

System Message: ERROR/3 (<stdin>, line 254)

Unknown directive type "autofunction".

.. autofunction:: scrapy.utils.asyncio.create_looping_call

System Message: ERROR/3 (<stdin>, line 255)

Unknown directive type "autoclass".

.. autoclass:: scrapy.utils.asyncio.AsyncioLoopingCall

System Message: ERROR/3 (<stdin>, line 256)

Unknown directive type "autofunction".

.. autofunction:: scrapy.utils.asyncio.run_in_thread

If your code needs to know whether the reactor is available, you can either check for the value of the :setting:`TWISTED_REACTOR_ENABLED` setting (you need access to the :class:`~scrapy.crawler.Crawler` instance to do this) or use the following function:

System Message: ERROR/3 (<stdin>, line 258); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 258); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 263)

Unknown directive type "autofunction".

.. autofunction:: scrapy.utils.reactorless.is_reactorless

In general, code that doesn't use the reactor (directly or indirectly) can be used unmodified both with the asyncio reactor and without a reactor. This includes code that converts Deferreds to futures and vice versa as described in :ref:`asyncio-await-dfd`.

System Message: ERROR/3 (<stdin>, line 265); backlink

Unknown interpreted text role "ref".

Troubleshooting

ImportError: Import of twisted.internet.reactor is forbidden when running without a Twisted reactor [...]: Scrapy is configured to run without a reactor, but some code imported :mod:`twisted.internet.reactor`, most likely because that code needs a reactor to be used. You need to stop using this code or set :setting:`TWISTED_REACTOR_ENABLED` back to True. It's also possible that the reactor isn't really needed but was installed due to the problem described in :ref:`asyncio-preinstalled-reactor`, in which case it should be enough to fix the problematic imports.

System Message: ERROR/3 (<stdin>, line 273); backlink

Unknown interpreted text role "mod".

System Message: ERROR/3 (<stdin>, line 273); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 273); backlink

Unknown interpreted text role "ref".

RuntimeError: TWISTED_REACTOR_ENABLED is False but a Twisted reactor is installed: Scrapy is configured to run without a reactor, but a reactor is already installed before the Scrapy code is executed. If you are trying to set :setting:`TWISTED_REACTOR_ENABLED` via :ref:`per-spider settings <spider-settings>`, it's currently unsupported.

System Message: ERROR/3 (<stdin>, line 282); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 282); backlink

Unknown interpreted text role "ref".

RuntimeError: We expected a Twisted reactor to be installed but it isn't: Scrapy is configured to run with a reactor and not to install one, but a reactor wasn't installed before the Scrapy code is executed. If you are trying to set :setting:`TWISTED_REACTOR_ENABLED` via :ref:`per-spider settings <spider-settings>`, it's currently unsupported.

System Message: ERROR/3 (<stdin>, line 288); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 288); backlink

Unknown interpreted text role "ref".

RuntimeError: <class> doesn't support TWISTED_REACTOR_ENABLED=False: The listed class cannot be used with :setting:`TWISTED_REACTOR_ENABLED` set to False. There may be a replacement in the :ref:`documentation above <asyncio-without-reactor>` or the documentation of the affected class.

System Message: ERROR/3 (<stdin>, line 294); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 294); backlink

Unknown interpreted text role "ref".

Windows-specific notes

The Windows implementation of :mod:`asyncio` can use two event loop implementations, :class:`~asyncio.ProactorEventLoop` (default) and :class:`~asyncio.SelectorEventLoop`. However, only :class:`~asyncio.SelectorEventLoop` works with Twisted.

System Message: ERROR/3 (<stdin>, line 305); backlink

Unknown interpreted text role "mod".

System Message: ERROR/3 (<stdin>, line 305); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 305); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 305); backlink

Unknown interpreted text role "class".

Scrapy changes the event loop class to :class:`~asyncio.SelectorEventLoop` automatically when you change the :setting:`TWISTED_REACTOR` setting or call :func:`~scrapy.utils.reactor.install_reactor`.

System Message: ERROR/3 (<stdin>, line 310); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 310); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 310); backlink

Unknown interpreted text role "func".

Note

Other libraries you use may require :class:`~asyncio.ProactorEventLoop`, e.g. because it supports subprocesses (this is the case with playwright), so you cannot use them together with Scrapy on Windows (but you should be able to use them on WSL or native Linux).

System Message: ERROR/3 (<stdin>, line 314); backlink

Unknown interpreted text role "class".

Note

This problem doesn't apply when not using the reactor, see :ref:`asyncio-without-reactor`.

System Message: ERROR/3 (<stdin>, line 320); backlink

Unknown interpreted text role "ref".

Using custom asyncio loops

You can also use custom asyncio event loops with the asyncio reactor. Set the :setting:`ASYNCIO_EVENT_LOOP` setting to the import path of the desired event loop class to use it instead of the default asyncio event loop.

System Message: ERROR/3 (<stdin>, line 331); backlink

Unknown interpreted text role "setting".

Switching to a non-asyncio reactor

If for some reason your code doesn't work with the asyncio reactor, you can use a different reactor by setting the :setting:`TWISTED_REACTOR` setting to its import path (e.g. 'twisted.internet.epollreactor.EPollReactor') or to None, which will use the default reactor for your platform. If you are using :class:`~scrapy.crawler.AsyncCrawlerRunner` or :class:`~scrapy.crawler.AsyncCrawlerProcess` you also need to switch to their Deferred-based counterparts: :class:`~scrapy.crawler.CrawlerRunner` or :class:`~scrapy.crawler.CrawlerProcess` respectively.

System Message: ERROR/3 (<stdin>, line 341); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 341); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 341); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 341); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 341); backlink

Unknown interpreted text role "class".
</html>