RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/docs/topics/components.rst

2.7 KiB
Raw Blame History

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> </head>

Components

A Scrapy component is any class whose objects are created using :func:`scrapy.utils.misc.create_instance`.

System Message: ERROR/3 (<stdin>, line 7); backlink

Unknown interpreted text role "func".

That includes the classes that you may assign to the following settings:

Third-party Scrapy components may also let you define additional Scrapy components, usually configurable through :ref:`settings <topics-settings>`, to modify their behavior.

System Message: ERROR/3 (<stdin>, line 40); backlink

Unknown interpreted text role "ref".

Enforcing component requirements

Sometimes, your components may only be intended to work under certain conditions. For example, they may require a minimum version of Scrapy to work as intended, or they may require certain settings to have specific values.

In addition to describing those conditions in the documentation of your component, it is a good practice to raise an exception from the __init__ method of your component if those conditions are not met at run time.

In the case of :ref:`downloader middlewares <topics-downloader-middleware>`, :ref:`extensions <topics-extensions>`, :ref:`item pipelines <topics-item-pipeline>`, and :ref:`spider middlewares <topics-spider-middleware>`, you should raise :exc:`scrapy.exceptions.NotConfigured`, passing a description of the issue as a parameter to the exception so that it is printed in the logs, for the user to see. For other components, feel free to raise whatever other exception feels right to you; for example, :exc:`RuntimeError` would make sense for a Scrapy version mismatch, while :exc:`ValueError` may be better if the issue is the value of a setting.

System Message: ERROR/3 (<stdin>, line 57); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 57); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 57); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 57); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 57); backlink

Unknown interpreted text role "exc".

System Message: ERROR/3 (<stdin>, line 57); backlink

Unknown interpreted text role "exc".

System Message: ERROR/3 (<stdin>, line 57); backlink

Unknown interpreted text role "exc".

If your requirement is a minimum Scrapy version, you may use :attr:`scrapy.__version__` to enforce your requirement. For example:

System Message: ERROR/3 (<stdin>, line 68); backlink

Unknown interpreted text role "attr".

System Message: WARNING/2 (<stdin>, line 71)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    from pkg_resources import parse_version

    import scrapy


    class MyComponent:
        def __init__(self):
            if parse_version(scrapy.__version__) < parse_version("2.7"):
                raise RuntimeError(
                    f"{MyComponent.__qualname__} requires Scrapy 2.7 or "
                    f"later, which allow defining the process_spider_output "
                    f"method of spider middlewares as an asynchronous "
                    f"generator."
                )
</html>