RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/docs/topics/components.rst

5.7 KiB
Raw Permalink Blame History

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> </head>

Components

A Scrapy component is any class whose objects are built using :func:`~scrapy.utils.misc.build_from_crawler`.

System Message: ERROR/3 (<stdin>, line 7); backlink

Unknown interpreted text role "func".

That includes the classes that you may assign to the following settings:

Third-party Scrapy components may also let you define additional Scrapy components, usually configurable through :ref:`settings <topics-settings>`, to modify their behavior.

System Message: ERROR/3 (<stdin>, line 44); backlink

Unknown interpreted text role "ref".

Initializing from the crawler

Any Scrapy component may optionally define the following class method:

System Message: ERROR/3 (<stdin>, line 55)

Unknown directive type "classmethod".

.. classmethod:: from_crawler(cls, crawler: scrapy.crawler.Crawler, *args, **kwargs)

    Return an instance of the component based on *crawler*.

    *args* and *kwargs* are component-specific arguments that some components
    receive. However, most components do not get any arguments, and instead
    :ref:`use settings <component-settings>`.

    If a component class defines this method, this class method is called to
    create any instance of the component.

    The *crawler* object provides access to all Scrapy core components like
    :ref:`settings <topics-settings>` and :ref:`signals <topics-signals>`,
    allowing the component to access them and hook its functionality into
    Scrapy.

Settings

Components can be configured through :ref:`settings <topics-settings>`.

System Message: ERROR/3 (<stdin>, line 76); backlink

Unknown interpreted text role "ref".

Components can read any setting from the :attr:`~scrapy.crawler.Crawler.settings` attribute of the :class:`~scrapy.crawler.Crawler` object they can :ref:`get for initialization <from-crawler>`. That includes both built-in and custom settings.

System Message: ERROR/3 (<stdin>, line 78); backlink

Unknown interpreted text role "attr".

System Message: ERROR/3 (<stdin>, line 78); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 78); backlink

Unknown interpreted text role "ref".

For example:

System Message: WARNING/2 (<stdin>, line 85)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    class MyExtension:
        @classmethod
        def from_crawler(cls, crawler):
            settings = crawler.settings
            return cls(settings.getbool("LOG_ENABLED"))

        def __init__(self, log_is_enabled=False):
            if log_is_enabled:
                print("log is enabled!")

Components do not need to declare their custom settings programmatically. However, they should document them, so that users know they exist and how to use them.

It is a good practice to prefix custom settings with the name of the component, to avoid collisions with custom settings of other existing (or future) components. For example, an extension called WarcCaching could prefix its custom settings with WARC_CACHING_.

Another good practice, mainly for components meant for :ref:`component priority dictionaries <component-priority-dictionaries>`, is to provide a boolean setting called <PREFIX>_ENABLED (e.g. WARC_CACHING_ENABLED) to allow toggling that component on and off without changing the component priority dictionary setting. You can usually check the value of such a setting during initialization, and if False, raise :exc:`~scrapy.exceptions.NotConfigured`.

System Message: ERROR/3 (<stdin>, line 106); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 106); backlink

Unknown interpreted text role "exc".

When choosing a name for a custom setting, it is also a good idea to have a look at the names of :ref:`built-in settings <topics-settings-ref>`, to try to maintain consistency with them.

System Message: ERROR/3 (<stdin>, line 114); backlink

Unknown interpreted text role "ref".

Enforcing requirements

Sometimes, your components may only be intended to work under certain conditions. For example, they may require a minimum version of Scrapy to work as intended, or they may require certain settings to have specific values.

In addition to describing those conditions in the documentation of your component, it is a good practice to raise an exception from the __init__ method of your component if those conditions are not met at run time.

In the case of :ref:`downloader middlewares <topics-downloader-middleware>`, :ref:`extensions <topics-extensions>`, :ref:`item pipelines <topics-item-pipeline>`, and :ref:`spider middlewares <topics-spider-middleware>`, you should raise :exc:`~scrapy.exceptions.NotConfigured`, passing a description of the issue as a parameter to the exception so that it is printed in the logs, for the user to see. For other components, feel free to raise whatever other exception feels right to you; for example, :exc:`RuntimeError` would make sense for a Scrapy version mismatch, while :exc:`ValueError` may be better if the issue is the value of a setting.

System Message: ERROR/3 (<stdin>, line 131); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 131); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 131); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 131); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 131); backlink

Unknown interpreted text role "exc".

System Message: ERROR/3 (<stdin>, line 131); backlink

Unknown interpreted text role "exc".

System Message: ERROR/3 (<stdin>, line 131); backlink

Unknown interpreted text role "exc".

If your requirement is a minimum Scrapy version, you may use :attr:`scrapy.__version__` to enforce your requirement. For example:

System Message: ERROR/3 (<stdin>, line 142); backlink

Unknown interpreted text role "attr".

System Message: WARNING/2 (<stdin>, line 145)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    from packaging.version import parse as parse_version

    import scrapy


    class MyComponent:
        def __init__(self):
            if parse_version(scrapy.__version__) < parse_version("2.7"):
                raise RuntimeError(
                    f"{MyComponent.__qualname__} requires Scrapy 2.7 or "
                    f"later, which allow defining the process_spider_output "
                    f"method of spider middlewares as an asynchronous "
                    f"generator."
                )

API reference

The following function can be used to create an instance of a component class:

System Message: ERROR/3 (<stdin>, line 167)

Unknown directive type "autofunction".

.. autofunction:: scrapy.utils.misc.build_from_crawler

The following function can also be useful when implementing a component, to report the import path of the component class, e.g. when reporting problems:

System Message: ERROR/3 (<stdin>, line 172)

Unknown directive type "autofunction".

.. autofunction:: scrapy.utils.python.global_object_name
</html>