RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/docs/topics/addons.rst

13 KiB
Raw Blame History

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> </head>

Add-ons

Scrapy's add-on system is a framework which unifies managing and configuring components that extend Scrapy's core functionality, such as middlewares, extensions, or pipelines. It provides users with a plug-and-play experience in Scrapy extension management, and grants extensive configuration control to developers.

Activating and configuring add-ons

Add-ons and their configuration live in Scrapy's :class:`~scrapy.addons.AddonManager`. During Scrapy's start-up process, and only then, the add-on manager will read a list of enabled add-ons and their configurations from your settings. There are two places where you can provide the paths to add-ons you want to enable:

System Message: ERROR/3 (<stdin>, line 17); backlink

Unknown interpreted text role "class".
  • the INSTALLED_ADDONS setting, and
  • the scrapy.cfg file.

As Scrapy settings can be modified from many places, e.g. in a project's settings.py, in a Spider's custom_settings attribute, or from the command line, using the INSTALLED_ADDONS setting is the preferred way to manage add-ons.

The INSTALLED_ADDONS setting a tuple in which every item is a path to an add-on. The path can be both a Python or a file path. While more precise, it is not necessary to specify the full add-on Python path if it is either built into Scrapy or lives in your project's addons submodule.

The configuration of an add-on, if necessary at all, is stored as a dictionary setting whose name is the uppercase add-on name.

This is an example where an internal add-on and two third-party add-ons (in this case with one requiring no configuration) are enabled/configured in a project's settings.py:

INSTALLED_ADDONS = (
    'httpcache',
    'path.to.some.addon',
    'path/to/other/addon.py',
)

HTTPCACHE = {
    'expiration_secs': 60,
    'ignore_http_codes': [404, 405],
}

SOMEADDON = {
    'some_config': True,
}

It is also possible to manage add-ons from scrapy.cfg. While the syntax is a little friendlier, be aware that this file, and therefore the configuration in it, is not bound to a particular Scrapy project. While this should not pose a problem when you use the project on your development machine only, a common stumbling block is that scrapy.cfg is not deployed via scrapyd-deploy.

In scrapy.cfg, section names, prepended with addon:, replace the dictionary keys. I.e., the configuration from above would look like this:

System Message: WARNING/2 (<stdin>, line 67)

Cannot analyze code. Pygments package not found.

.. code-block:: cfg

    [addon:httpcache]
    expiration_secs = 60
    ignore_http_codes = 404,405

    [addon:path.to.some.addon]
    some_config = true

    [addon:path/to/other/addon.py]

Enabling and configuring add-ons within Python code

The :class:`~scrapy.addons.AddonManager` will only read from Scrapy's settings and from scrapy.cfg at the beginning of Scrapy's start-up process. Afterwards, i.e. as soon as the :class:`~scrapy.addons.AddonManager` is populated, changing the INSTALLED_ADDONS setting or any of the add-on configuration dictionary settings will have no effect.

System Message: ERROR/3 (<stdin>, line 82); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 82); backlink

Unknown interpreted text role "class".

If you want to enable, disable, or configure add-ons in Python code, for example when writing your own add-on, you will have to use the :class:`~scrapy.addons.AddonManager`. You can access the add-on manager through either crawler.addons or, if you are writing an add-on, through the addons argument of the :meth:`update_addons` callback. The add-on manager provides many useful methods and attributes to facilitate interacting with the add-ons framework, e.g.:

System Message: ERROR/3 (<stdin>, line 88); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 88); backlink

Unknown interpreted text role "meth".

In this example, we ensure that the httpcache add-on is loaded, and that its expiration_secs configuration is set to 60:

# addons is an instance of AddonManager
if 'httpcache' not in addons:
    addons.add('httpcache', {'expiration_secs': 60})
else:
    addons.configs['httpcache']['expiration_secs'] = 60

Writing your own add-ons

Add-ons are (any) Python objects that provide Scrapy's add-on interface. The interface is enforced through zope.interface. This leaves the choice of Python object up the developer. Examples:

  • for a small pipeline, the add-on interface could be implemented in the same class that also implements the open/close_spider and process_item callbacks
  • for larger add-ons, or for clearer structure, the interface could be provided by a stand-alone module

The absolute minimum interface consists of two attributes:

System Message: ERROR/3 (<stdin>, line 128)

Unknown directive type "attribute".

.. attribute:: name

    string with add-on name

System Message: ERROR/3 (<stdin>, line 132)

Unknown directive type "attribute".

.. attribute:: version

    version string (PEP-404, e.g. ``'1.0.1'``)

Of course, stating just these two attributes will not get you very far. Add-ons can provide three callback methods that are called at various stages before the crawling process:

System Message: ERROR/3 (<stdin>, line 140)

Unknown directive type "method".

.. method:: update_settings(config, settings)

    This method is called during the initialization of the
    :class:`~scrapy.crawler.Crawler`. Here, you should perform dependency checks
    (e.g. for external Python libraries) and update the
    :class:`~scrapy.settings.Settings` object as wished, e.g. enable components
    for this add-on or set required configuration of other extensions.

    :param config: Configuration of this add-on
    :type config: ``dict``

    :param settings: The settings object storing Scrapy/component configuration
    :type settings: :class:`~scrapy.settings.Settings`

System Message: ERROR/3 (<stdin>, line 154)

Unknown directive type "method".

.. method:: check_configuration(config, crawler)

    This method is called when the :class:`~scrapy.crawler.Crawler` has been
    fully initialized, immediately before it starts crawling. You can perform
    additional dependency and configuration checks here.

    :param config: Configuration of this add-on
    :type config: ``dict``

    :param crawler: Fully initialized Scrapy crawler
    :type crawler: :class:`~scrapy.crawler.Crawler`

System Message: ERROR/3 (<stdin>, line 166)

Unknown directive type "method".

.. method:: update_addons(config, addons)

    This method is called immediately before :meth:`update_settings`, and should
    be used to enable and configure other *add-ons* only.

    When using this callback, be aware that there is no guarantee in which order
    the :meth:`update_addons` callbacks of enabled add-ons will be called.
    Add-ons that are added to the :class:`~scrapy.addons.AddonManager` during
    this callback will also have their :meth:`update_addons` method called.

    :param config: Configuration of this add-on
    :type config: ``dict``

    :param addons: Add-on manager holding all loaded add-ons
    :type addons: :class:`~scrapy.addons.AddonManager`

Additionally, add-ons may (and should, where appropriate) provide one or more attributes that can be used for limited automated detection of possible dependency clashes:

System Message: ERROR/3 (<stdin>, line 186)

Unknown directive type "attribute".

.. attribute:: requires

    list of built-in or custom components needed by this add-on, as strings.

System Message: ERROR/3 (<stdin>, line 190)

Unknown directive type "attribute".

.. attribute:: modifies

    list of built-in or custom components whose functionality is affected or
    replaced by this add-on (a custom HTTP cache should list ``httpcache`` here)

System Message: ERROR/3 (<stdin>, line 195)

Unknown directive type "attribute".

.. attribute:: provides

    list of components provided by this add-on (e.g. ``mongodb`` for an
    extension that provides generic read/write access to a MongoDB database)

The entries in the :attr:`requires` and :attr:`modifies` attributes can be add-on names or components from other add-ons' :attr:`provides` attribute. You can specify PEP 440-style information about required versions. Examples:

System Message: ERROR/3 (<stdin>, line 200); backlink

Unknown interpreted text role "attr".

System Message: ERROR/3 (<stdin>, line 200); backlink

Unknown interpreted text role "attr".

System Message: ERROR/3 (<stdin>, line 200); backlink

Unknown interpreted text role "attr".
requires = ['httpcache']
requires = ['otheraddon >= 2.0', 'yetanotheraddon']

The Python object or module that is pointed to by an add-on path (e.g. given in the INSTALLED_ADDONS setting, or given to :meth:`~scrapy.addons.AddonManager.add`) does not necessarily have to be an add-on. Instead, it can provide an _addon attribute. This attribute can be either an add-on or another add-on path.

System Message: ERROR/3 (<stdin>, line 207); backlink

Unknown interpreted text role "meth".

Add-on base class

Scrapy comes with a built-in base class for add-ons which provides some convenience functionality:

By default, the base add-on class will expose the add-on configuration into Scrapy's settings namespace, in caps and with the add-on name prepended. It is easy to write your own functionality while still being able to use the convenience functions by overwriting :meth:`~scrapy.addons.Addon.update_settings`.

System Message: ERROR/3 (<stdin>, line 235); backlink

Unknown interpreted text role "meth".

System Message: ERROR/3 (<stdin>, line 241)

Unknown directive type "module".

.. module:: scrapy.addons

System Message: ERROR/3 (<stdin>, line 243)

Unknown directive type "autoclass".

.. autoclass:: Addon
   :members:

Add-on examples

Set some basic configuration using the :class:`Addon` base class:

System Message: ERROR/3 (<stdin>, line 250); backlink

Unknown interpreted text role "class".
from scrapy.addons import Addon

class MyAddon(Addon):
    name = 'myaddon'
    version = '1.0'
    component = 'path.to.mypipeline'
    component_type = 'ITEM_PIPELINES'
    component_order = 200
    basic_settings = {
        'DNSCACHE_ENABLED': False,
    }

Check dependencies:

from scrapy.addons import Addon

class MyAddon(Addon):
    name = 'myaddon'
    version = '1.0'

    def update_settings(self, config, settings):
        try:
            import boto
        except ImportError:
            raise RuntimeError("myaddon requires the boto library")
        else:
            self.export_config(config, settings)

Enable a component that lives relative to the add-on (see :ref:`topics-api-settings`):

System Message: ERROR/3 (<stdin>, line 280); backlink

Unknown interpreted text role "ref".
from scrapy.addons import Addon

class MyAddon(Addon):
    name = 'myaddon'
    version = '1.0'
    component = __name__ + '.downloadermw.coolmw'
    component_type = 'DOWNLOADER_MIDDLEWARES'
    component_order = 900

Instantiate components ad hoc:

from path.to.my.pipelines import MySQLPipeline

class MyAddon(object):
    name = 'myaddon'
    version = '1.0'

    def update_settings(self, config, settings):
        mysqlpl = MySQLPipeline(password=config['password'])
        settings.set(
            'ITEM_PIPELINES',
            {mysqlpl: 200},
            priority='addon',
        )

Provide add-on interface along component interface:

class MyPipeline(object):
    name = 'mypipeline'
    version = '1.0'

    def process_item(self, item, spider):
        # Do some processing here
        return item

    def update_settings(self, config, settings):
        settings.set(
            'ITEM_PIPELINES',
            {self: 200},
            priority='addon',
        )

Enable another addon (see :ref:`topics-api-addonmanager`):

System Message: ERROR/3 (<stdin>, line 325); backlink

Unknown interpreted text role "ref".
class MyAddon(object):
    name = 'myaddon'
    version = '1.0'

    def update_addons(self, config, addons):
        if 'httpcache' not in addons.enabled:
            addons.add('httpcache', {'expiration_secs': 60})

Check configuration of fully initialized crawler (see :ref:`topics-api-crawler`):

System Message: ERROR/3 (<stdin>, line 335); backlink

Unknown interpreted text role "ref".
class MyAddon(object):
    name = 'myaddon'
    version = '1.0'

    def update_settings(self, config, settings):
        settings.set('DNSCACHE_ENABLED', False, priority='addon')

    def check_configuration(self, config, crawler):
        if crawler.settings.getbool('DNSCACHE_ENABLED'):
            # The spider, some other add-on, or the user messed with the
            # DNS cache setting
            raise ValueError("myaddon is incompatible with DNS cache")

Provide add-on interface through a module:

System Message: WARNING/2 (<stdin>, line 354)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    name = 'AddonModule'
    version = '1.0'

    class MyPipeline(object):
        # ...

    class MyDownloaderMiddleware(object):
        # ...

    def update_settings(config, settings):
        settings.set(
            'ITEM_PIPELINES',
            {MyPipeline(): 200},
            priority='addon',
            }
        settings.set(
            'DOWNLOADER_MIDDLEWARES',
            {MyDownloaderMiddleware(): 800},
            priority='addon',
            }

Forward to other add-ons depending on Python version:

# This could be a Python module, say project/pipelines/mypipeline.py, but
# could also be done inside a class, etc.
import six

if six.PY3:
    # We're running Python 3
    _addon = 'path.to.addon'
else:
    _addon = 'path.to.other.addon'
</html>