diff --git a/sep/sep-021.rst b/sep/sep-021.rst deleted file mode 100644 index 47cba004c..000000000 --- a/sep/sep-021.rst +++ /dev/null @@ -1,324 +0,0 @@ -======= =================== -SEP 21 -Title Add-ons -Author Pablo Hoffman -Created 2014-02-14 -Status Draft -======= =================== - -================ -SEP-021: Add-ons -================ - -This proposal introduces add-ons, a unified way to manage Scrapy extensions, -middlewares and pipelines. - -Scrapy currently supports many hooks and mechanisms for extending its -functionality, but no single entry point for enabling and configuring them. -Instead, the hooks are spread over: - -* Spider middlewares (``SPIDER_MIDDLEWARES``) -* Downloader middlewares (``DOWNLOADER_MIDDLEWARES``) -* Downloader handlers (``DOWNLOADER_HANDLERS``) -* Item pipelines (``ITEM_PIPELINES``) -* Feed exporters and storages (``FEED_EXPORTERS``, ``FEED_STORAGES``) -* Overridable components (``DUPEFILTER_CLASS``, ``STATS_CLASS``, - ``SCHEDULER``, ``SPIDER_MANAGER_CLASS``, ``ITEM_PROCESSOR``, etc.) -* Generic extensions (``EXTENSIONS``) -* CLI commands (``COMMANDS_MODULE``) - -This approach has several shortfalls: - -* Enabling an extension often requires modifying many settings, often in a - coordinated way, which is complex and error prone. -* Extension developers have little control over ensuring their library - dependencies and configuration requirements are met, especially since most - extensions never 'see' a fully-configured crawler before it starts running. -* The user is burdened with supervising potential interplay of extensions, - especially non-included ones, ranging from setting name clashes to mutually - excluding dependencies/configuration requirements. - -*Add-ons* search to remedy these shortcomings by enhancing Scrapy's extension -management, making it easy-to-use and transparent for users while giving more -configuration control to developers. - - -Design goals and non-goals -========================== - -Goals: - -* simple to manage: adding or removing extensions should be just a matter of - adding or removing lines in a configuration file -* backward compatibility with enabling extension the "old way" (i.e. modifying - settings directly) - -Non-goals: - -* a way to publish, distribute or discover extensions (use pypi for that) - - -User experience: managing add-ons -================================= - -Add-ons are enabled and configured either via Scrapy's settings, or (for add-ons -not bound to any project) in ``scrapy.cfg``. - -In the settings, add-ons can be enabled by adding either their name (for -built-in add-ons), their Python path, or their file path, to a -``INSTALLED_ADDONS`` setting. If necessary, each add-on can be configured by -providing a dictionary-valued setting with the uppercase add-on name. For -example, to enable and configure the built-in ``httpcache`` add-on and enable -(without configuring) two custom add-ons, one via Python path and one via file -path, add these entries to your settings module:: - - INSTALLED_ADDONS = ( - 'httpcache', - 'mymodule.filters.myfilter', - 'mymodule/filters/otherfilter.py', - ) - - HTTPCACHE = { - 'ignore_http_codes': [404, 503], - } - -In ``scrapy.cfg``, add-ons are enabled and configured with one section per -add-on. The section names correspond to the entries of ``INSTALLED_ADDONS``. -The configuration from above could look like this:: - - [addon:httpcache] - ignore_http_codes = 404,503 - - [addon:mymodule.filters.myfilter] - - [addon:mymodule/filters/otherfilter.py] - - -Developer experience: writing add-ons -===================================== - -Add-ons are (any) Python *objects* that implement Scrapy's *add-on interface*. -The interface is enforced through ``zope.interface``. This leaves the choice of -Python object up the developer. Examples: - -* for a small pipeline, the add-on interface could be implemented in the same - class that also implements the ``open/close_spider`` and ``process_item`` - callbacks -* for larger add-ons, or for clearer structure, the interface could be provided - by a stand-alone module - -The absolute minimum interface consists of just two attributes: - -* ``NAME``: string with add-on name -* ``VERSION``: PEP-440 style version string - -To be any useful, an add-on should implement at least one of the following -callback methods: - -* ``update_addons()``: adds and configures other add-ons -* ``update_settings()``: sets configuration (such as default values for this - add-on and required settings for other extensions) and enables needed - components. -* ``check_configuration()``: receives the fully-initialized ``Crawler`` - instance before it starts running, performs additional dependency and - configuration requirement checks - -Additionally, an add-on may (and should, where appropriate) provide one or more -variables that can be used for automated detection of possible dependency -clashes: - -* ``REQUIRES``: list of built-in or custom components required by this add-on, - as PEP-440 strings -* ``MODIFIES``: list of components whose functionality is affected or replaced - by this add-on (a custom HTTP cache should list ``httpcache`` here) -* ``PROVIDES``: list of components provided by this add-on (e.g. ``mongodb`` - for an extension that provides generic read/write access to a MongoDB - database, releasing other components from having to provide their own - database access methods) - -update_addons() ------------------ - -Called: -~~~~~~~ - -Shortly after initialisation of the ``Crawler`` object. - -Arguments: -~~~~~~~~~~ - -* ``config``: configuration of this add-on -* ``addons``: the add-on manager, providing methods to add and configure add-ons - -Purpose: -~~~~~~~~ - -* Configure and enable related add-ons, useful for 'umbrella add-ons' which - chain-load other add-ons based on the configuration - -Examples: -~~~~~~~~~ - -.. code-block:: python - - def update_addons(config, addons): - if "httpcache" not in addons.enabled: - addons.add("httpcache", {"expiration_secs": 60}) - -or: - -.. code-block:: python - - def update_addons(config, addons): - if "otheraddon" in addons.enabled: - addons.configs["otheraddon"]["some_config_name"] = True - -update_settings() ------------------ - -Called: -~~~~~~~ - -Directly after the ``update_addons()`` callback of all add-ons has been called. - -Arguments: -~~~~~~~~~~ - -* ``config``: configuration of this add-on -* ``settings``: the crawler's ``Settings`` instance containing all project - settings - -Purpose: -~~~~~~~~ - -* Modify ``settings`` to enable required components -* Expose some add-on specific configuration (``config``) into the global - settings namespace (``settings``) if necessary -* Raise exception if components can not be properly configured (e.g. on missing - dependencies); Scrapy will print this exception *and exit* (making users - explicitly acknowledge that the add-on does not work by forcing them to - disable it). - -Side note: -~~~~~~~~~~ - -The ``MiddlewareManager.from_settings()`` method will receive a slight -modification to allow directly placing Python objects instead of class paths -in the middleware dict settings. This way, add-ons can place already -instantiated components into the settings. This allows keeping configuration -as local to components as possible and avoids cluttering up the global -settings namespace. Furthermore, it allows reusing components (e.g. using -two instances of the same mongodb pipeline to write to different locations). - -Examples: -~~~~~~~~~ - -:: - - def update_settings(config, settings): - # Don't care where this module is located - settings.set['DOWNLADER_MIDDLEWARES']({ - __name__ + '.downloadermw.coolmw': 900, - }) - - # Instantiate components to not expose settings into - # the global namespace - from .pipelines import MySQLPipeline - mysqlpl = MySQLPipeline(password = config['password']) - settings.set['ITEM_PIPELINES']({ - mysqlpl: 200, - }) - -or:: - - def update_settings(config, settings): - # Assuming this class also has a process_item() method - settings.set['ITEM_PIPELINES']({ - self: 200, - }) - -or:: - - def update_settings(config, settings): - try: - import boto - except ImportError: - raise RuntimeError("boto library is required") - -check_configuration() ---------------------- - -Called: -~~~~~~~ - -Shortly before the crawler starts crawling. - -Arguments: -~~~~~~~~~~ - -* ``config``: configuration of this add-on -* ``crawler``: fully-initialized ``Crawler`` object, ready to start crawling - -Purpose: -~~~~~~~~ - -* Perform post-initialization checks like making sure the extension and its - dependencies were configured properly. -* Raise exception if a critical check failed; Scrapy will print this exception - *and exit* (see ``update_settings()`` purpose for rationale on this). - -Examples: -~~~~~~~~~ - -:: - - def check_configuration(config, crawler): - if 'some.other.addon' not in crawler.addons.enabled: - raise RuntimeError("Some other add-on required to use this add-on") - - -Implementation -============== - -A new core component, the *add-on manager*, is introduced to Scrapy. It -facilitates loading add-ons, gathering and providing information on them, -calling their callbacks at appropriate times, and performing basic checks for -dependency and configuration clashes. - -Layout ------- - -A new ``AddonManager`` class is introduced, providing methods to - -* add and remove add-ons, -* search for add-ons by name -* read enabled add-ons and their configurations from the settings module and - from ``settings.py``, -* enable and disable add-ons -* check for possible dependency incompatibilites by inspecting the collected - ``REQUIRES``, ``MODIFIES`` and ``PROVIDES`` add-on variables -* call the add-on callbacks - -Integration into start-up process ---------------------------------- - -The settings used to crawl are not complete until the spider-specific settings -have been loaded in ``Crawler.__init__()``. Add-on management follows this -approach and only starts loading add-ons when the crawler is initialised. - -Instantiation and the calls ``update_addons()`` and ``update_settings()`` happen -in ``Crawler.__init__()``. The final checks (i.e. the callback to -``check_configuration()``) is coded into the ``Crawler.crawl()`` method after -creating the engine. - -Finding add-ons ---------------- - -Add-on localisation is governed by the add-on paths given in -``INSTALLED_ADDONS`` (or by the section names if using ``scrapy.cfg``). If -nothing is found at the given path, it is tried again with ``addons.`` -prepended (i.e. pointing to the project's ``addons`` folder or module), then -with ``scrapy.addons.`` prepended (i.e. pointing to Scrapy's ``addons`` -submodule). If the object found has an ``_addon`` attribute, that attribute -will be treated as the found add-on. This allows, for example, to change the -add-on based on the Python version.