Remove unneeded code.

docs/topics/addons.rst

@@ -15,12 +15,13 @@ Activating and configuring add-ons
 ==================================
 
 Add-ons and their configuration live in Scrapy's
-:class:`~scrapy.addons.AddonManager`. During Scrapy's start-up process, and
+:class:`~scrapy.addons.AddonManager`. During a :class:`~scrapy.crawler.Crawler`
-only then, the add-on manager will read a list of enabled add-ons and their
+initialization the add-on manager will read a list of enabled add-ons from your
-configurations from your ``ADDONS`` setting.
+``ADDONS`` setting and their optional configuration from the respective
+settings.
 
 The ``ADDONS`` setting is a dict in which every key is an addon class or its
-import path and the vaoue is its priority.
+import path and the value is its priority.
 
 The configuration of an add-on, if necessary at all, is stored as a dictionary
 setting whose name is the uppercase add-on name.
@@ -37,66 +38,17 @@ configuration) are enabled/configured in a project's ``settings.py``::
         'some_config': True,
     }
 
-Enabling and configuring add-ons within Python code
----------------------------------------------------
-
-The :class:`~scrapy.addons.AddonManager` will only read from Scrapy's settings
-*at the beginning* of Scrapy's start-up process.
-Afterwards, i.e. as soon as the :class:`~scrapy.addons.AddonManager` is
-populated, changing the ``ADDONS`` setting or any of the add-on
-configuration dictionary settings will have no effect.
-
-If you want to enable, disable, or configure add-ons in Python code, for example
-when writing your own add-on, you will have to use the
-:class:`~scrapy.addons.AddonManager`. You can access the add-on manager through
-either ``crawler.addons`` or, if you are writing an add-on, through the
-``addons`` argument of the :meth:`update_addons` callback. The add-on manager
-provides many useful methods and attributes to facilitate interacting with the
-add-ons framework, e.g.:
-
-* an :meth:`~scrapy.addons.AddonManager.add` method to load add-ons,
-* the :attr:`~scrapy.addons.AddonManager.enabled` list of enabled add-ons,
-* :meth:`~scrapy.addons.AddonManager.enable` and
-  :meth:`~scrapy.addons.AddonManager.disable` methods,
-* the :attr:`~scrapy.addons.AddonManager.configs` dictionary which holds the
-  configuration of all add-ons.
-
-In this example, we ensure that the ``httpcache`` add-on is loaded, and that
-its ``expiration_secs`` configuration is set to ``60``::
-
-    # addons is an instance of AddonManager
-    if 'httpcache' not in addons:
-        addons.add('httpcache', {'expiration_secs': 60})
-    else:
-        addons.configs['httpcache']['expiration_secs'] = 60
-
 
 Writing your own add-ons
 ========================
 
-Add-ons are (any) Python *objects* that provide Scrapy's *add-on interface*.
+Add-ons are (any) Python *objects* that provide Scrapy's *add-on interface*:
-The interface is enforced through ``zope.interface``. This leaves the choice of
-Python object up the developer. Examples:
-
-* for a small pipeline, the add-on interface could be implemented in the same
-  class that also implements the ``open/close_spider`` and ``process_item``
-  callbacks,
-* for larger add-ons, or for clearer structure, the interface could be provided
-  by a stand-alone module.
-
-The absolute minimum interface consists of two attributes:
 
 .. attribute:: name
 
     string with add-on name
 
-.. attribute:: version
+    :type: ``str``
-
-    version string (PEP-404, e.g. ``'1.0.1'``)
-
-Of course, stating just these two attributes will not get you very far. Add-ons
-can provide three callback methods that are called at various stages before the
-crawling process:
 
 .. method:: update_settings(config, settings)
 
@@ -124,75 +76,18 @@ crawling process:
     :param crawler: Fully initialized Scrapy crawler
     :type crawler: :class:`~scrapy.crawler.Crawler`
 
-.. method:: update_addons(config, addons)
-
-    This method is called immediately before :meth:`update_settings`, and should
-    be used to enable and configure other *add-ons* only.
-
-    Add-ons that are added to the :class:`~scrapy.addons.AddonManager` during
-    this callback will also have their :meth:`update_addons` method called.
-
-    :param config: Configuration of this add-on
-    :type config: ``dict``
-
-    :param addons: Add-on manager holding all loaded add-ons
-    :type addons: :class:`~scrapy.addons.AddonManager`
-
-Additionally, add-ons may (and should, where appropriate) provide one or more
-attributes that can be used for limited automated detection of possible
-dependency clashes:
-
-.. attribute:: requires
-
-    list of built-in or custom components needed by this add-on, as strings.
-
-.. attribute:: modifies
-
-    list of built-in or custom components whose functionality is affected or
-    replaced by this add-on (a custom HTTP cache should list ``httpcache`` here)
-
-.. attribute:: provides
-
-    list of components provided by this add-on (e.g. ``mongodb`` for an
-    extension that provides generic read/write access to a MongoDB database)
-
-The entries in the :attr:`requires` and :attr:`modifies` attributes can be add-on
-names or components from other add-ons' :attr:`provides` attribute. You can
-specify :pep:`440`-style information about required versions. Examples::
-
-    requires = ['httpcache']
-    requires = ['otheraddon >= 2.0', 'yetanotheraddon']
-
-The Python object or module that is pointed to by an add-on path (e.g. given in
-the ``ADDONS`` setting, or given to
-:meth:`~scrapy.addons.AddonManager.add`) does not necessarily have to be an
-add-on. Instead, it can provide an ``_addon`` attribute. This attribute can be
-either an add-on or another add-on path.
-
 
 Add-on base class
 =================
 
 Scrapy comes with a built-in base class for add-ons which provides some
-convenience functionality:
+convenience functionality: the add-on configuration can be exposed into
-
+Scrapy's settings via :meth:`~scrapy.addons.Addon.export_config`, configurable
-* basic settings can be exported via :meth:`~scrapy.addons.Addon.export_basics`,
+via :attr:`~scrapy.addons.Addon.default_config` and
-  configurable via :attr:`~scrapy.addons.Addon.basic_settings`.
+:attr:`~scrapy.addons.Addon.config_mapping`.
-* a single component (e.g. an item pipeline or a downloader middleware) can be
-  inserted into Scrapy's settings via
-  :meth:`~scrapy.addons.Addon.export_component`, configurable via
-  :attr:`~scrapy.addons.Addon.component_type`,
-  :attr:`~scrapy.addons.Addon.component_key`,
-  :attr:`~scrapy.addons.Addon.component`, and the ``order`` key in
-  :attr:`~scrapy.addons.Addon.default_config`.
-* the add-on configuration can be exposed into Scrapy's settings via
-  :meth:`~scrapy.addons.Addon.export_config`, configurable via
-  :attr:`~scrapy.addons.Addon.default_config`,
-  :attr:`~scrapy.addons.Addon.config_mapping`, and
-  :attr:`~scrapy.addons.Addon.settings_prefix`.
 
 By default, the base add-on class will expose the add-on configuration into
-Scrapy's settings namespace, in caps and with the add-on name prepended. It is
+Scrapy's settings namespace, in upper case. It is
 easy to write your own functionality while still being able to use the
 convenience functions by overwriting
 :meth:`~scrapy.addons.Addon.update_settings`.
@@ -213,13 +108,11 @@ Set some basic configuration using the :class:`Addon` base class::
 
     class MyAddon(Addon):
         name = 'myaddon'
-        version = '1.0'
+
-        component = 'path.to.mypipeline'
+        def update_settings(self, config, settings):
-        component_type = 'ITEM_PIPELINES'
+            super().update_settings(settings)
-        component_order = 200
+            settings["ITEM_PIPELINES"]["path.to.mypipeline"] = 200
-        basic_settings = {
+            settings["DNSCACHE_ENABLED"] = True
-            'DNSCACHE_ENABLED': False,
-        }
 
 Check dependencies::
 
@@ -227,78 +120,22 @@ Check dependencies::
 
     class MyAddon(Addon):
         name = 'myaddon'
-        version = '1.0'
 
         def update_settings(self, config, settings):
             try:
                 import boto
             except ImportError:
                 raise RuntimeError("myaddon requires the boto library")
-            self.export_config(config, settings)
+            super().update_settings(settings)
-
-Enable a component that lives relative to the add-on (see
-:ref:`topics-api-settings`)::
-
-    from scrapy.addons import Addon
-
-    class MyAddon(Addon):
-        name = 'myaddon'
-        version = '1.0'
-        component = __name__ + '.downloadermw.coolmw'
-        component_type = 'DOWNLOADER_MIDDLEWARES'
-        component_order = 900
-
-Instantiate components ad hoc::
-
-    from path.to.my.pipelines import MySQLPipeline
-
-    class MyAddon(object):
-        name = 'myaddon'
-        version = '1.0'
-
-        def update_settings(self, config, settings):
-            mysqlpl = MySQLPipeline(password=config['password'])
-            settings.set(
-                'ITEM_PIPELINES',
-                {mysqlpl: 200},
-                priority='addon',
-            )
-
-Provide add-on interface along component interface::
-
-    class MyPipeline(object):
-        name = 'mypipeline'
-        version = '1.0'
-
-        def process_item(self, item, spider):
-            # Do some processing here
-            return item
-
-        def update_settings(self, config, settings):
-            settings.set(
-                'ITEM_PIPELINES',
-                {self: 200},
-                priority='addon',
-            )
-
-Enable another addon (see :ref:`topics-api-addonmanager`)::
-
-    class MyAddon(object):
-        name = 'myaddon'
-        version = '1.0'
-
-        def update_addons(self, config, addons):
-            if 'httpcache' not in addons.enabled:
-                addons.add('httpcache', {'expiration_secs': 60})
 
 Check configuration of fully initialized crawler (see
 :ref:`topics-api-crawler`)::
 
     class MyAddon(object):
         name = 'myaddon'
-        version = '1.0'
 
         def update_settings(self, config, settings):
+            super().update_settings(settings)
             settings.set('DNSCACHE_ENABLED', False, priority='addon')
 
         def check_configuration(self, config, crawler):
@@ -306,43 +143,3 @@ Check configuration of fully initialized crawler (see
                 # The spider, some other add-on, or the user messed with the
                 # DNS cache setting
                 raise ValueError("myaddon is incompatible with DNS cache")
-
-Provide add-on interface through a module:
-
-.. code-block:: python
-
-    name = "AddonModule"
-    version = "1.0"
-
-
-    class MyPipeline(object):
-        ...
-
-
-    class MyDownloaderMiddleware(object):
-        ...
-
-
-    def update_settings(config, settings):
-        settings.set(
-            "ITEM_PIPELINES",
-            {MyPipeline(): 200},
-            priority="addon",
-        )
-        settings.set(
-            "DOWNLOADER_MIDDLEWARES",
-            {MyDownloaderMiddleware(): 800},
-            priority="addon",
-        )
-
-Forward to other add-ons depending on Python version::
-
-    # This could be a Python module, say project/pipelines/mypipeline.py, but
-    # could also be done inside a class, etc.
-    import six
-
-    if six.PY3:
-        # We're running Python 3
-        _addon = 'path.to.addon'
-    else:
-        _addon = 'path.to.other.addon'

scrapy/addons.py

@@ -1,108 +1,27 @@
-import warnings
-from collections import OrderedDict, defaultdict
 from collections.abc import Mapping
-from inspect import isclass
+from typing import Any, Dict, Iterator, Optional, OrderedDict
-from typing import Dict
 
-import zope.interface
-from packaging.requirements import Requirement
-from packaging.version import Version
-from zope.interface.verify import verifyObject
-
-from scrapy.interfaces import IAddon
 from scrapy.utils.conf import build_component_list
-from scrapy.utils.misc import load_module_or_object
+from scrapy.utils.misc import load_object
 
 
-@zope.interface.implementer(IAddon)
 class Addon(object):
-    basic_settings = None
+    name: str
-    """``dict`` of settings that will be exported via :meth:`export_basics`."""
 
     default_config = None
     """``dict`` with default configuration."""
 
     config_mapping = None
     """``dict`` with mappings from config names to setting names. The given
-    setting names will be taken as given, i.e. they will be neither prefixed
+    setting names will be taken as given, not uppercased.
-    nor uppercased.
     """
 
-    component_type = None
-    """Component setting into which to export via :meth:`export_component`. Can
-    be any of the dictionary-like component setting names. If ``None``,
-    :meth:`export_component` will do nothing.
-    """
-
-    component_key = None
-    """Key to be used in the component dictionary setting when exporting via
-    :meth:`export_component`. This is only useful for the settings that have
-    no order, e.g. ``DOWNLOAD_HANDLERS`` or ``FEED_EXPORTERS``.
-    """
-
-    component_order = 0
-    """Component order to use when not given in the add-on configuration. Has
-    no effect for component types that use :attr:`component_key`.
-    """
-
-    component = None
-    """Component to be inserted via :meth:`export_component`. This can be
-    anything that can be used in the dictionary-like component settings, i.e.
-    a class path or a class. If ``None``, it is assumed that the add-on itself
-    also provides the component interface, and ``self`` will be used.
-    """
-
-    settings_prefix = None
-    """Prefix with which the add-on configuration will be exported into the
-    global settings namespace via :meth:`export_config`. If ``None``,
-    :attr:`name` will be used. If ``False``, no configuration will be exported.
-    """
-
-    def export_component(self, config, settings):
-        """Export the component in :attr:`component` into the dictionary-like
-        component setting derived from :attr:`component_type`.
-
-        Where applicable, the order parameter of the component (i.e. the
-        dictionary value) will be retrieved from the ``order`` add-on
-        configuration value.
-
-        :param config: Add-on configuration from which to read component order
-        :type config: ``dict``
-
-        :param settings: Settings object into which to export component
-        :type settings: :class:`~scrapy.settings.Settings`
-        """
-        if self.component_type:
-            comp = self.component or self
-            if self.component_key:
-                # e.g. for DOWNLOAD_HANDLERS: {'http': 'myclass'}
-                k = self.component_key
-                v = comp
-            else:
-                # e.g. for DOWNLOADER_MIDDLEWARES: {'myclass': 100}
-                k = comp
-                v = config.get("order", self.component_order)
-            settings[self.component_type].update({k: v}, "addon")
-
-    def export_basics(self, settings):
-        """Export the :attr:`basic_settings` attribute into the settings object.
-
-        All settings will be exported with ``addon`` priority (see
-        :ref:`topics-api-settings`).
-
-        :param settings: Settings object into which to expose the basic settings
-        :type settings: :class:`~scrapy.settings.Settings`
-        """
-        for setting, value in (self.basic_settings or {}).items():
-            settings.set(setting, value, "addon")
-
     def export_config(self, config, settings):
-        """Export the add-on configuration, all keys in caps and with
+        """Export the add-on configuration, all keys in caps, into the settings
-        :attr:`settings_prefix` or :attr:`name` prepended, into the settings
         object.
 
         For example, the add-on configuration ``{'key': 'value'}`` will export
-        the setting ``ADDONNAME_KEY`` with a value of ``value``. All settings
+        the setting ``KEY`` with a value of ``value``. All settings
         will be exported with ``addon`` priority (see
         :ref:`topics-api-settings`).
 
@@ -112,11 +31,8 @@ class Addon(object):
         :param settings: Settings object into which to export the configuration
         :type settings: :class:`~scrapy.settings.Settings`
         """
-        if self.settings_prefix is False:
-            return
         conf = self.default_config or {}
         conf.update(config)
-        prefix = self.settings_prefix or self.name
         # Since default exported config is case-insensitive (everything will be
         # uppercased), make mapped config case-insensitive as well
         conf_mapping = {k.lower(): v for k, v in (self.config_mapping or {}).items()}
@@ -124,14 +40,11 @@ class Addon(object):
             if key.lower() in conf_mapping:
                 key = conf_mapping[key.lower()]
             else:
-                key = (prefix + "_" + key).upper()
+                key = key.upper()
             settings.set(key, val, "addon")
 
     def update_settings(self, config, settings):
-        """Export both the basic settings and the add-on configuration. I.e.,
+        """Modifiy `settings` to enable and configure required components.
-        call :meth:`export_basics` and :meth:`export_config`.
-
-        For more advanced add-ons, you may want to override this callback.
 
         :param config: Add-on configuration
         :type config: ``dict``
@@ -139,12 +52,21 @@ class Addon(object):
         :param settings: Crawler settings object
         :type settings: :class:`~scrapy.settings.Settings`
         """
-        self.export_component(config, settings)
-        self.export_basics(settings)
         self.export_config(config, settings)
 
+    def check_configuration(self, config, crawler):
+        """Perform post-initialization checks on fully configured `crawler`.
 
-class AddonManager(Mapping):
+        :param config: Add-on configuration
+        :type config: ``dict``
+
+        :param crawler: the fully-initialized crawler
+        :type crawler: :class:`~scrapy.crawler.Crawler`
+        """
+        pass
+
+
+class AddonManager(Mapping[str, Addon]):
     """This class facilitates loading and storing :ref:`topics-addons`.
 
     You can treat it like a read-only dictionary in which keys correspond to
@@ -160,108 +82,42 @@ class AddonManager(Mapping):
 
     """
 
-    def __init__(self):
+    def __init__(self) -> None:
-        self._addons = OrderedDict()
+        self._addons: OrderedDict[str, Addon] = OrderedDict[str, Addon]()
-        self.configs = {}
+        self.configs: Dict[str, Dict[str, Any]] = {}
-        self._disable_on_add = []
 
-    def __getitem__(self, name):
+    def __getitem__(self, name: str) -> Addon:
         return self._addons[name]
 
-    def __delitem__(self, name):
+    def __iter__(self) -> Iterator[str]:
-        del self._addons[name]
-        del self.configs[name]
-
-    def __iter__(self):
         return iter(self._addons)
 
-    def __len__(self):
+    def __len__(self) -> int:
         return len(self._addons)
 
-    def add(self, addon, config=None):
+    def add(self, addon: Any, config: Optional[Dict[str, Any]] = None):
         """Store an add-on.
 
-        If ``addon`` is a string, it will be treated as add-on path and passed
-        to :meth:`get_addon`. Otherwise, ``addon`` must be a Python object
-        implementing or providing Scrapy's add-on interface. The interface
-        will be enforced through ``zope.interface``'s ``verifyObject()``.
-
-        If ``addon`` is a class, it will be instantiated. You can avoid this
-        (for example if you have implemented the add-on callbacks as class
-        methods) by declaring --  via ``zope.interface`` -- that your class
-        directly *provides* ``scrapy.interfaces.IAddon``.
-
         :param addon: The add-on object (or path) to be stored
-        :type addon: Any Python object providing the add-on interface or ``str``
+        :type addon: Python object, class or ``str``
 
         :param config: The add-on configuration dictionary
         :type config: ``dict``
         """
-        addon = self.get_addon(addon)
+        if isinstance(addon, (type, str)):
-        if isclass(addon) and not IAddon.providedBy(addon):
+            addon = load_object(addon)
+        if isinstance(addon, type):
             addon = addon()
-        if not IAddon.providedBy(addon):
-            zope.interface.alsoProvides(addon, IAddon)
-        # zope.interface's exceptions are already quite helpful. Still, should
-        # we catch them and log an error message?
-        verifyObject(IAddon, addon)
         name = addon.name
         if name in self:
             raise ValueError(f"Addon '{name}' already loaded")
         self._addons[name] = addon
         self.configs[name] = config or {}
-        if name in self._disable_on_add:
-            self.configs[name]["_enabled"] = False
-            self._disable_on_add.remove(name)
-
-    def remove(self, addon):
-        """Remove an add-on.
-
-        If ``addon`` is the name of a stored add-on, that add-on will be
-        removed. Otherwise, you can use the argument in the same fashion as
-        in :meth:`add`.
-
-        :param addon: The add-on name, object, or path to be removed
-        :type addon: Any Python object providing the add-on interface or ``str``
-        """
-        if addon in self:
-            del self[addon]
-        elif hasattr(addon, "name") and addon.name in self:
-            del self[addon.name]
-        else:
-            try:
-                del self[self.get_addon(addon).name]
-            except NameError:
-                raise KeyError
-
-    @staticmethod
-    def get_addon(path):
-        """Get an add-on object by its Python or file path.
-
-        ``path`` is assumed to be an import path of an add-on. If the object or
-        module pointed to by ``path`` has an attribute named ``_addon`` that
-        attribute will be assumed to be the add-on. :meth:`get_addon` will keep
-        following ``_addon`` attributes until it finds an object that does not
-        have an attribute named ``_addon``.
-
-        :param path: Import path of an add-on
-        :type path: ``str``
-        """
-        if isinstance(path, str):
-            try:
-                obj = load_module_or_object(path)
-            except NameError:
-                raise NameError(f"Could not find add-on '{path}'")
-        else:
-            obj = path
-        if hasattr(obj, "_addon"):
-            obj = AddonManager.get_addon(obj._addon)
-        return obj
 
     def load_settings(self, settings):
         """Load add-ons and configurations from settings object.
 
-        This will invoke :meth:`get_addon` for every add-on path in the
+        This will load the addon for every add-on path in the
         ``ADDONS`` setting. For each of these add-ons, the configuration will be
         read from the dictionary setting whose name matches the uppercase add-on
         name.
@@ -271,141 +127,12 @@ class AddonManager(Mapping):
         :type settings: :class:`~scrapy.settings.Settings`
         """
         paths = build_component_list(settings["ADDONS"])
-        addons = [self.get_addon(path) for path in paths]
+        addons = [load_object(path) for path in paths]
         configs = [settings.getdict(addon.name.upper()) for addon in addons]
         for a, c in zip(addons, configs):
             self.add(a, c)
 
-    def check_dependency_clashes(self) -> None:
+    def update_settings(self, settings) -> None:
-        """Check for incompatibilities in add-on dependencies.
-
-        Add-ons can provide information about their dependencies in their
-        ``provides``, ``modifies`` and ``requires`` attributes. This method will
-        raise an ``ImportError`` if
-
-        * a component required by an add-on is not provided by any other add-on,
-          or
-        * a component modified by an add-on is not provided by any other add-on,
-          or
-        * the same component is provided by more than one add-on,
-
-        and warn when a component required by an add-on is modified by any other
-        add-on.
-        """
-        # Collect all active add-ons and the components they provide
-        versions: Dict[str, Version] = {}
-
-        def add_version(project_name, version):
-            if project_name in versions:
-                raise ImportError(
-                    f"Component {project_name} provided by multiple add-ons"
-                )
-            versions[project_name] = Version(version)
-
-        for name in self:
-            ver = self[name].version
-            add_version(name, ver)
-            for provides_name in getattr(self[name], "provides", []):
-                add_version(provides_name, ver)
-
-        # Collect all required and modified components
-        def compile_attribute_dict(attribute_name):
-            attrs = defaultdict(list)
-            for name in self:
-                for entry in getattr(self[name], attribute_name, []):
-                    attrs[entry].append(name)
-            return attrs
-
-        modified = compile_attribute_dict("modifies")
-        required = compile_attribute_dict("requires")
-
-        req_or_mod = set(required.keys()).union(modified.keys())
-        for reqstr in req_or_mod:
-            req = Requirement(reqstr)
-            if req.name not in versions or versions[req.name] not in req.specifier:
-                raise ImportError(
-                    f"Add-ons {required[reqstr] + modified[reqstr]} require"
-                    f" or modify missing component {reqstr}"
-                )
-
-        mod_and_req = set(required.keys()).intersection(modified.keys())
-        for conflict in mod_and_req:
-            warnings.warn(
-                f"Component '{conflict}', required by add-ons {required[conflict]},"
-                f" is modified by add-ons {modified[conflict]}"
-            )
-
-    def disable(self, addon):
-        """Disable an add-on, i.e. prevent its callbacks from being called.
-
-        If you disable an add-on before it is loaded, it will be disabled as
-        soon as it is added to the :class:`AddonManager`.
-
-        :param addon: Name of the add-on to be disabled
-        :type addon: ``str``
-        """
-        if addon in self:
-            self.configs[addon]["_enabled"] = False
-        else:
-            self._disable_on_add.append(addon)
-
-    def enable(self, addon):
-        """Re-enable a disabled add-on.
-
-        Will raise ``ValueError`` if the add-on is neither already loaded nor
-        marked for being disabled on adding.
-
-        :param addon: Name of the add-on to be enabled
-        :type addon: ``str``
-        """
-        if addon in self:
-            self.configs[addon]["_enabled"] = True
-        elif addon in self._disable_on_add:
-            self._disable_on_add.remove(addon)
-        else:
-            raise ValueError("Add-ons need to be added before they can be enabled")
-
-    @property
-    def disabled(self):
-        """Names of disabled add-ons"""
-        return [
-            a for a in self if not self.configs[a].get("_enabled", True)
-        ] + self._disable_on_add
-
-    @property
-    def enabled(self):
-        """Names of enabled add-ons"""
-        return [a for a in self if self.configs[a].get("_enabled", True)]
-
-    def _call_if_exists(self, obj, cbname, *args, **kwargs):
-        if obj is None:
-            return
-        try:
-            cb = getattr(obj, cbname)
-        except AttributeError:
-            return
-        else:
-            cb(*args, **kwargs)
-
-    def _call_addon(self, addonname, cbname, *args, **kwargs):
-        if self.configs[addonname].get("_enabled", True):
-            self._call_if_exists(
-                self[addonname], cbname, self.configs[addonname], *args, **kwargs
-            )
-
-    def update_addons(self):
-        """Call ``update_addons()`` of all held add-ons.
-
-        This will also call ``update_addons()`` of all add-ons that are added
-        last minute during the ``update_addons()`` routine of other add-ons.
-        """
-        called_addons = set()
-        while called_addons != set(self):
-            for name in set(self).difference(called_addons):
-                called_addons.add(name)
-                self._call_addon(name, "update_addons", self)
-
-    def update_settings(self, settings):
         """Call ``update_settings()`` of all held add-ons.
 
         :param settings: The :class:`~scrapy.settings.Settings` object to be \
@@ -413,13 +140,13 @@ class AddonManager(Mapping):
         :type settings: :class:`~scrapy.settings.Settings`
         """
         for name in self:
-            self._call_addon(name, "update_settings", settings)
+            self[name].update_settings(self.configs[name], settings)
 
-    def check_configuration(self, crawler):
+    def check_configuration(self, crawler) -> None:
         """Call ``check_configuration()`` of all held add-ons.
 
         :param crawler: the fully-initialized crawler
         :type crawler: :class:`~scrapy.crawler.Crawler`
         """
         for name in self:
-            self._call_addon(name, "check_configuration", crawler)
+            self[name].check_configuration(self.configs[name], crawler)

scrapy/cmdline.py

@@ -6,7 +6,6 @@ import sys
 from importlib.metadata import entry_points
 
 import scrapy
-from scrapy.addons import AddonManager
 from scrapy.commands import BaseRunSpiderCommand, ScrapyCommand, ScrapyHelpFormatter
 from scrapy.crawler import CrawlerProcess
 from scrapy.exceptions import UsageError
@@ -131,8 +130,6 @@ def execute(argv=None, settings=None):
         else:
             settings["EDITOR"] = editor
 
-    addons = AddonManager()
-
     inproject = inside_project()
     cmds = _get_commands_dict(settings, inproject)
     cmdname = _pop_command_name(argv)
@@ -156,7 +153,7 @@ def execute(argv=None, settings=None):
     opts, args = parser.parse_known_args(args=argv[1:])
     _run_print_help(parser, cmd.process_options, args, opts)
 
-    cmd.crawler_process = CrawlerProcess(settings, addons=addons)
+    cmd.crawler_process = CrawlerProcess(settings)
     _run_print_help(parser, _run_command, cmd, args, opts)
     sys.exit(cmd.exitcode)
 

scrapy/crawler.py

@@ -58,7 +58,6 @@ class Crawler:
         spidercls: Type[Spider],
         settings: Union[None, dict, Settings] = None,
         init_reactor: bool = False,
-        addons: Optional[AddonManager] = None,
     ):
         if isinstance(spidercls, Spider):
             raise ValueError("The spidercls argument must be a class, not an object")
@@ -70,10 +69,8 @@ class Crawler:
         self.settings: Settings = settings.copy()
         self.spidercls.update_settings(self.settings)
 
-        self.addons: AddonManager = addons if addons is not None else AddonManager()
+        self.addons: AddonManager = AddonManager()
         self.addons.load_settings(self.settings)
-        self.addons.update_addons()
-        self.addons.check_dependency_clashes()
         self.addons.update_settings(self.settings)
 
         self.signals: SignalManager = SignalManager(self)
@@ -200,15 +197,10 @@ class CrawlerRunner:
             )
         return loader_cls.from_settings(settings.frozencopy())
 
-    def __init__(
+    def __init__(self, settings: Union[Dict[str, Any], Settings, None] = None):
-        self,
-        settings: Union[Dict[str, Any], Settings, None] = None,
-        addons: Optional[AddonManager] = None,
-    ):
         if isinstance(settings, dict) or settings is None:
             settings = Settings(settings)
         self.settings = settings
-        self.addons: Optional[AddonManager] = addons
         self.spider_loader = self._get_spider_loader(settings)
         self._crawlers: Set[Crawler] = set()
         self._active: Set[defer.Deferred] = set()
@@ -292,7 +284,7 @@ class CrawlerRunner:
     def _create_crawler(self, spidercls: Union[str, Type[Spider]]) -> Crawler:
         if isinstance(spidercls, str):
             spidercls = self.spider_loader.load(spidercls)
-        return Crawler(spidercls, self.settings, addons=self.addons)
+        return Crawler(spidercls, self.settings)
 
     def stop(self):
         """
@@ -338,13 +330,8 @@ class CrawlerProcess(CrawlerRunner):
     process. See :ref:`run-from-script` for an example.
     """
 
-    def __init__(
+    def __init__(self, settings=None, install_root_handler=True):
-        self,
+        super().__init__(settings)
-        settings=None,
-        install_root_handler: bool = True,
-        addons: Optional[AddonManager] = None,
-    ):
-        super().__init__(settings, addons)
         configure_logging(self.settings, install_root_handler)
         log_scrapy_info(self.settings)
         self._initialized_reactor = False

scrapy/interfaces.py

@@ -1,4 +1,4 @@
-from zope.interface import Attribute, Interface
+from zope.interface import Interface
 
 
 class ISpiderLoader(Interface):
@@ -15,22 +15,3 @@ class ISpiderLoader(Interface):
 
     def find_by_request(request):
         """Return the list of spiders names that can handle the given request"""
-
-
-class IAddon(Interface):
-    """Scrapy add-on"""
-
-    name = Attribute("""Add-on name""")
-    version = Attribute("""Add-on version string (PEP440)""")
-
-    # XXX: Can methods be declared optional? I.e., can I enforce the signature
-    #      but not the existence of a method?
-
-    # def update_addons(config, addons):
-    #    """Enables and configures other add-ons"""
-
-    # def update_settings(config, settings):
-    #    """Modifies `settings` to enable and configure required components"""
-
-    # def check_configuration(config, crawler):
-    #    """Performs post-initialization checks on fully configured `crawler`"""

scrapy/utils/conf.py

@@ -99,18 +99,12 @@ def init_env(project="default", set_syspath=True):
             sys.path.append(projdir)
 
 
-def config_from_filepath(sources):
-    """Create a ConfigParser and read in the given `sources`, which can be
-    either a filename or a list of filenames."""
-    cfg = ConfigParser()
-    cfg.read(sources)
-    return cfg
-
-
 def get_config(use_closest=True):
     """Get Scrapy config file as a ConfigParser"""
     sources = get_sources(use_closest)
-    return config_from_filepath(sources)
+    cfg = ConfigParser()
+    cfg.read(sources)
+    return cfg
 
 
 def get_sources(use_closest=True) -> List[str]:

scrapy/utils/misc.py

@@ -67,7 +67,7 @@ def load_object(path: Union[str, Callable]) -> Any:
         if callable(path):
             return path
         raise TypeError(
-            "Unexpected argument type, expected string " f"or object, got: {type(path)}"
+            f"Unexpected argument type, expected string or object, got: {type(path)}"
         )
 
     try:
@@ -86,22 +86,6 @@ def load_object(path: Union[str, Callable]) -> Any:
     return obj
 
 
-def load_module_or_object(path):
-    """Load python module or (non-module) object from given path.
-
-    Path can be both a Python or a file path.
-    """
-    try:
-        return import_module(path)
-    except ImportError:
-        pass
-    try:
-        return load_object(path)
-    except (ValueError, NameError, ImportError):
-        pass
-    raise NameError(f"Could not load '{path}'")
-
-
 def walk_modules(path: str) -> List[ModuleType]:
     """Loads a module and all its submodules from the given module path and
     returns them. If *any* module throws an exception while importing, that

scrapy/utils/test.py

@@ -73,7 +73,6 @@ def get_crawler(
     spidercls: Optional[Type[Spider]] = None,
     settings_dict: Optional[Dict[str, Any]] = None,
     prevent_warnings: bool = True,
-    addons=None,
 ) -> Crawler:
     """Return an unconfigured Crawler object. If settings_dict is given, it
     will be used to populate the crawler settings with a project level
@@ -87,7 +86,7 @@ def get_crawler(
     if prevent_warnings:
         settings["REQUEST_FINGERPRINTER_IMPLEMENTATION"] = "2.7"
     settings.update(settings_dict or {})
-    runner = CrawlerRunner(settings, addons)
+    runner = CrawlerRunner(settings)
     return runner.create_crawler(spidercls or Spider)
 
 

tests/test_addons.py

@@ -0,0 +1,72 @@
+import unittest
+
+from scrapy.addons import Addon, AddonManager
+from scrapy.settings import BaseSettings
+
+
+class GoodAddon(object):
+    name = "GoodAddon"
+
+    def update_settings(self, config, settings):
+        pass
+
+    def check_configuration(self, config, crawler):
+        pass
+
+
+class AddonTest(unittest.TestCase):
+    def setUp(self):
+        class AddonWithAttributes(Addon):
+            name = "Test"
+
+        self.testaddon = AddonWithAttributes()
+
+    def test_export_config(self):
+        settings = BaseSettings()
+        self.testaddon.config_mapping = {"MAPPED_key": "MAPPING_WORKED"}
+        self.testaddon.default_config = {"key": 55, "defaultkey": 100}
+        self.testaddon.export_config(
+            {"key": 313, "OTHERKEY": True, "mapped_KEY": 99}, settings
+        )
+        self.assertEqual(settings["KEY"], 313)
+        self.assertEqual(settings["DEFAULTKEY"], 100)
+        self.assertEqual(settings["OTHERKEY"], True)
+        self.assertNotIn("MAPPED_key", settings)
+        self.assertNotIn("MAPPED_KEY", settings)
+        self.assertEqual(settings["MAPPING_WORKED"], 99)
+        self.assertEqual(settings.getpriority("KEY"), 15)
+
+    def test_update_settings(self):
+        settings = BaseSettings()
+        settings.set("KEY1", "default", priority="default")
+        settings.set("KEY2", "project", priority="project")
+        addon_config = {"key1": "addon", "key2": "addon", "key3": "addon"}
+        self.testaddon.update_settings(addon_config, settings)
+        self.assertEqual(settings["KEY1"], "addon")
+        self.assertEqual(settings["KEY2"], "project")
+        self.assertEqual(settings["KEY3"], "addon")
+
+
+class AddonManagerTest(unittest.TestCase):
+    def setUp(self):
+        self.manager = AddonManager()
+
+    def test_add(self):
+        manager = AddonManager()
+        manager.add("tests.test_addons.GoodAddon")
+        self.assertCountEqual(manager, ["GoodAddon"])
+        self.assertIsInstance(manager["GoodAddon"], GoodAddon)
+
+    def test_load_settings(self):
+        settings = BaseSettings()
+        settings.set(
+            "ADDONS",
+            {"tests.test_addons.GoodAddon": 0},
+        )
+        settings.set("GOODADDON", {"key": "val2"})
+        manager = AddonManager()
+        manager.load_settings(settings)
+        self.assertCountEqual(manager, ["GoodAddon"])
+        self.assertIsInstance(manager["GoodAddon"], GoodAddon)
+        self.assertCountEqual(manager.configs["GoodAddon"], ["key"])
+        self.assertEqual(manager.configs["GoodAddon"]["key"], "val2")

tests/test_addons/__init__.py

@@ -1,342 +0,0 @@
-import itertools
-import unittest
-import warnings
-from unittest import mock
-
-from zope.interface import directlyProvides
-from zope.interface.exceptions import BrokenImplementation, MultipleInvalid
-from zope.interface.verify import verifyObject
-
-from scrapy.addons import Addon, AddonManager
-from scrapy.crawler import Crawler
-from scrapy.interfaces import IAddon
-from scrapy.settings import BaseSettings
-
-from . import addonmod, addons
-
-
-class AddonTest(unittest.TestCase):
-    def setUp(self):
-        self.rawaddon = Addon()
-
-        class AddonWithAttributes(Addon):
-            name = "Test"
-            version = "1.0"
-
-        self.testaddon = AddonWithAttributes()
-
-    def test_interface(self):
-        # Raw Addon should fail exactly b/c name and version are not given
-        self.assertFalse(hasattr(self.rawaddon, "name"))
-        self.assertFalse(hasattr(self.rawaddon, "version"))
-        self.assertRaises(MultipleInvalid, verifyObject, IAddon, self.rawaddon)
-        verifyObject(IAddon, self.testaddon)
-
-    def test_export_component(self):
-        settings = BaseSettings(
-            {"ITEM_PIPELINES": BaseSettings(), "DOWNLOAD_HANDLERS": BaseSettings()},
-            "default",
-        )
-        self.testaddon.component_type = None
-        self.testaddon.export_component({}, settings)
-        self.assertEqual(len(settings["ITEM_PIPELINES"]), 0)
-        self.testaddon.component_type = "ITEM_PIPELINES"
-        self.testaddon.component = "test.component"
-        self.testaddon.export_component({}, settings)
-        self.assertCountEqual(settings["ITEM_PIPELINES"], ["test.component"])
-        self.assertEqual(settings["ITEM_PIPELINES"]["test.component"], 0)
-        self.testaddon.component_order = 313
-        self.testaddon.export_component({}, settings)
-        self.assertEqual(settings["ITEM_PIPELINES"]["test.component"], 313)
-        self.testaddon.component_type = "DOWNLOAD_HANDLERS"
-        self.testaddon.component_key = "http"
-        self.testaddon.export_component({}, settings)
-        self.assertEqual(settings["DOWNLOAD_HANDLERS"]["http"], "test.component")
-
-    def test_export_basics(self):
-        settings = BaseSettings()
-        self.testaddon.basic_settings = {"TESTKEY": 313, "OTHERKEY": True}
-        self.testaddon.export_basics(settings)
-        self.assertEqual(settings["TESTKEY"], 313)
-        self.assertEqual(settings["OTHERKEY"], True)
-        self.assertEqual(settings.getpriority("TESTKEY"), 15)
-
-    def test_export_config(self):
-        settings = BaseSettings()
-        self.testaddon.settings_prefix = None
-        self.testaddon.config_mapping = {"MAPPED_key": "MAPPING_WORKED"}
-        self.testaddon.default_config = {"key": 55, "defaultkey": 100}
-        self.testaddon.export_config(
-            {"key": 313, "OTHERKEY": True, "mapped_KEY": 99}, settings
-        )
-        self.assertEqual(settings["TEST_KEY"], 313)
-        self.assertEqual(settings["TEST_DEFAULTKEY"], 100)
-        self.assertEqual(settings["TEST_OTHERKEY"], True)
-        self.assertNotIn("MAPPED_key", settings)
-        self.assertNotIn("MAPPED_KEY", settings)
-        self.assertEqual(settings["MAPPING_WORKED"], 99)
-        self.assertEqual(settings.getpriority("TEST_KEY"), 15)
-
-        self.testaddon.settings_prefix = "PREF"
-        self.testaddon.export_config({"newkey": 99}, settings)
-        self.assertEqual(settings["PREF_NEWKEY"], 99)
-
-        with mock.patch.object(settings, "set") as mock_set:
-            self.testaddon.settings_prefix = False
-            self.testaddon.export_config({"thirdnewkey": 99}, settings)
-            self.assertEqual(mock_set.call_count, 0)
-
-    def test_update_settings(self):
-        settings = BaseSettings()
-        settings.set("TEST_KEY1", "default", priority="default")
-        settings.set("TEST_KEY2", "project", priority="project")
-        self.testaddon.settings_prefix = None
-        self.testaddon.basic_settings = {"OTHERTEST_KEY": "addon"}
-        addon_config = {"key1": "addon", "key2": "addon", "key3": "addon"}
-        self.testaddon.update_settings(addon_config, settings)
-        self.assertEqual(settings["OTHERTEST_KEY"], "addon")
-        self.assertEqual(settings["TEST_KEY1"], "addon")
-        self.assertEqual(settings["TEST_KEY2"], "project")
-        self.assertEqual(settings["TEST_KEY3"], "addon")
-
-
-class AddonManagerTest(unittest.TestCase):
-    def setUp(self):
-        self.manager = AddonManager()
-
-    def test_add(self):
-        manager = AddonManager()
-        manager.add(addonmod, {"key": "val1"})
-        manager.add("tests.test_addons.addons.GoodAddon")
-        self.assertCountEqual(manager, ["AddonModule", "GoodAddon"])
-        self.assertIsInstance(manager["GoodAddon"], addons.GoodAddon)
-        self.assertCountEqual(manager.configs["AddonModule"], ["key"])
-        self.assertEqual(manager.configs["AddonModule"]["key"], "val1")
-        self.assertRaises(ValueError, manager.add, addonmod)
-
-    def test_add_dont_instantiate_providing_classes(self):
-        class ProviderGoodAddon(addons.GoodAddon):
-            pass
-
-        directlyProvides(ProviderGoodAddon, IAddon)
-        manager = AddonManager()
-        manager.add(ProviderGoodAddon)
-        self.assertIs(manager["GoodAddon"], ProviderGoodAddon)
-
-    def test_add_verifies(self):
-        brokenaddon = self.manager.get_addon("tests.test_addons.addons.BrokenAddon")
-        self.assertRaises(
-            BrokenImplementation,
-            self.manager.add,
-            brokenaddon,
-        )
-
-    def test_add_adds_missing_interface_declaration(self):
-        class GoodAddonWithoutDeclaration(object):
-            name = "GoodAddonWithoutDeclaration"
-            version = "1.0"
-
-        self.manager.add(GoodAddonWithoutDeclaration)
-
-    def test_remove(self):
-        manager = AddonManager()
-
-        def test_gets_removed(removearg):
-            manager.add(addonmod)
-            self.assertIn("AddonModule", manager)
-            manager.remove(removearg)
-            self.assertNotIn("AddonModule", manager)
-
-        test_gets_removed("AddonModule")
-        test_gets_removed(addonmod)
-        test_gets_removed("tests.test_addons.addonmod")
-        self.assertRaises(KeyError, manager.remove, "nonexistent")
-        self.assertRaises(KeyError, manager.remove, addons.GoodAddon())
-
-    def test_get_addon(self):
-        goodaddon = self.manager.get_addon("tests.test_addons.addons.GoodAddon")
-        self.assertIs(goodaddon, addons.GoodAddon)
-
-        loaded_addonmod = self.manager.get_addon("tests.test_addons.addonmod")
-        self.assertIs(loaded_addonmod, addonmod)
-
-        goodaddon = self.manager.get_addon("tests.test_addons.addons")
-        self.assertIsInstance(goodaddon, addons.GoodAddon)
-
-        self.assertRaises(NameError, self.manager.get_addon, "xy.n_onexistent")
-
-    def test_get_addon_forward(self):
-        class SomeCls(object):
-            _addon = "tests.test_addons.addons.GoodAddon"
-
-        self.assertIs(self.manager.get_addon(SomeCls()), addons.GoodAddon)
-
-    def test_get_addon_nested(self):
-        x = addons.GoodAddon("outer")
-        x._addon = addons.GoodAddon("middle")
-        x._addon._addon = addons.GoodAddon("inner")
-        self.assertIs(self.manager.get_addon(x), x._addon._addon)
-
-    def test_load_settings(self):
-        settings = BaseSettings()
-        settings.set(
-            "ADDONS",
-            {"tests.test_addons.addonmod": 0, "tests.test_addons.addons.GoodAddon": 0},
-        )
-        settings.set("ADDONMODULE", {"key": "val1"})
-        settings.set("GOODADDON", {"key": "val2"})
-        manager = AddonManager()
-        manager.load_settings(settings)
-        self.assertCountEqual(manager, ["GoodAddon", "AddonModule"])
-        self.assertIsInstance(manager["GoodAddon"], addons.GoodAddon)
-        self.assertCountEqual(manager.configs["GoodAddon"], ["key"])
-        self.assertEqual(manager.configs["GoodAddon"]["key"], "val2")
-        self.assertEqual(manager["AddonModule"], addonmod)
-        self.assertIn("key", manager.configs["AddonModule"])
-        self.assertEqual(manager.configs["AddonModule"]["key"], "val1")
-
-    def test_load_settings_order(self):
-        # Get three addons named 0, 1, 2
-        addonlist = [addons.GoodAddon(str(x)) for x in range(3)]
-        # Test for every possible ordering
-        for ordered_addons in itertools.permutations(addonlist):
-            expected_order = [a.name for a in ordered_addons]
-            settings = BaseSettings(
-                {"ADDONS": {a: i for i, a in enumerate(ordered_addons)}}
-            )
-            manager = AddonManager()
-            manager.load_settings(settings)
-            self.assertEqual(list(manager.keys()), expected_order)
-
-    def test_enabled_disabled(self):
-        manager = AddonManager()
-        manager.add(addons.GoodAddon("FirstAddon"))
-        manager.add(addons.GoodAddon("SecondAddon"))
-        self.assertEqual(set(manager.enabled), set(("FirstAddon", "SecondAddon")))
-        self.assertEqual(manager.disabled, [])
-        manager.disable("FirstAddon")
-        self.assertEqual(manager.enabled, ["SecondAddon"])
-        self.assertEqual(manager.disabled, ["FirstAddon"])
-        manager.enable("FirstAddon")
-        self.assertEqual(set(manager.enabled), set(("FirstAddon", "SecondAddon")))
-        self.assertEqual(manager.disabled, [])
-
-    def test_enable_before_add(self):
-        manager = AddonManager()
-        self.assertRaises(ValueError, manager.enable, "FirstAddon")
-        manager.disable("FirstAddon")
-        manager.enable("FirstAddon")
-        manager.add(addons.GoodAddon("FirstAddon"))
-        self.assertIn("FirstAddon", manager.enabled)
-
-    def test_disable_before_add(self):
-        manager = AddonManager()
-        manager.disable("FirstAddon")
-        manager.add(addons.GoodAddon("FirstAddon"))
-        self.assertEqual(manager.disabled, ["FirstAddon"])
-
-    def test_callbacks(self):
-        first_addon = addons.GoodAddon("FirstAddon")
-        second_addon = addons.GoodAddon("SecondAddon")
-
-        manager = AddonManager()
-        manager.add(first_addon, {"test": "first"})
-        manager.add(second_addon, {"test": "second"})
-        crawler = mock.create_autospec(Crawler)
-        settings = BaseSettings()
-
-        with mock.patch.object(
-            first_addon, "update_addons"
-        ) as ua_first, mock.patch.object(
-            second_addon, "update_addons"
-        ) as ua_second, mock.patch.object(
-            first_addon, "update_settings"
-        ) as us_first, mock.patch.object(
-            second_addon, "update_settings"
-        ) as us_second, mock.patch.object(
-            first_addon, "check_configuration"
-        ) as cc_first, mock.patch.object(
-            second_addon, "check_configuration"
-        ) as cc_second:
-            manager.update_addons()
-            ua_first.assert_called_once_with(manager.configs["FirstAddon"], manager)
-            ua_second.assert_called_once_with(manager.configs["SecondAddon"], manager)
-            manager.update_settings(settings)
-            us_first.assert_called_once_with(manager.configs["FirstAddon"], settings)
-            us_second.assert_called_once_with(manager.configs["SecondAddon"], settings)
-            manager.check_configuration(crawler)
-            cc_first.assert_called_once_with(manager.configs["FirstAddon"], crawler)
-            cc_second.assert_called_once_with(manager.configs["SecondAddon"], crawler)
-            self.assertEqual(ua_first.call_count, 1)
-            self.assertEqual(ua_second.call_count, 1)
-            self.assertEqual(us_first.call_count, 1)
-            self.assertEqual(us_second.call_count, 1)
-
-            us_first.reset_mock()
-            us_second.reset_mock()
-            manager.disable("FirstAddon")
-            manager.update_settings(settings)
-            self.assertEqual(us_first.call_count, 0)
-            manager.enable("FirstAddon")
-            manager.update_settings(settings)
-            self.assertEqual(us_first.call_count, 1)
-            self.assertEqual(us_second.call_count, 2)
-
-        # This will become relevant when we let spiders implement the add-on
-        # interface and should be replaced with a test where
-        # AddonManager.spidercls = None then.
-        manager._call_if_exists(None, "irrelevant")
-
-    def test_update_addons_last_minute_add(self):
-        class AddedAddon(addons.GoodAddon):
-            name = "AddedAddon"
-
-        class FirstAddon(addons.GoodAddon):
-            name = "FirstAddon"
-
-            def update_addons(self, config, addons):
-                addons.add(AddedAddon())
-
-        manager = AddonManager()
-        first_addon = FirstAddon()
-        with mock.patch.object(
-            first_addon, "update_addons", wraps=first_addon.update_addons
-        ) as ua_first, mock.patch.object(AddedAddon, "update_addons") as ua_added:
-            manager.add(first_addon, {"non-empty": "dict"})
-            manager.update_addons()
-            self.assertCountEqual(manager, ["FirstAddon", "AddedAddon"])
-            ua_first.assert_called_once_with(manager.configs["FirstAddon"], manager)
-            ua_added.assert_called_once_with(manager.configs["AddedAddon"], manager)
-
-    def test_check_dependency_clashes_attributes(self):
-        provides = addons.GoodAddon("ProvidesAddon")
-        provides.provides = ("test",)
-        provides2 = addons.GoodAddon("ProvidesAddon2")
-        provides2.provides = ("test",)
-        requires = addons.GoodAddon("RequiresAddon")
-        requires.requires = ("test",)
-        requires_name = addons.GoodAddon("RequiresNameAddon")
-        requires_name.requires = ("ProvidesAddon",)
-        requires_newer = addons.GoodAddon("RequiresNewerAddon")
-        requires_newer.requires = ("test>=2.0",)
-        modifies = addons.GoodAddon("ModifiesAddon")
-        modifies.modifies = ("test",)
-
-        def check_with(*addons):
-            manager = AddonManager()
-            for a in addons:
-                manager.add(a)
-            return manager.check_dependency_clashes()
-
-        self.assertRaises(ImportError, check_with, requires)
-        self.assertRaises(ImportError, check_with, modifies)
-        self.assertRaises(ImportError, check_with, provides, provides2)
-        self.assertRaises(ImportError, check_with, provides, requires_newer)
-        with warnings.catch_warnings(record=True) as w:
-            check_with(provides, modifies)
-            check_with(provides)
-            check_with(provides, requires)
-            check_with(provides, requires_name)
-            self.assertEqual(len(w), 0)
-            check_with(requires, provides, modifies)
-            self.assertEqual(len(w), 1)

tests/test_addons/addonmod.py

@@ -1,16 +0,0 @@
-from zope.interface import moduleProvides
-
-from scrapy.interfaces import IAddon
-
-moduleProvides(IAddon)
-
-name = "AddonModule"
-version = "1.0"
-
-
-def update_settings(config, settings):
-    pass
-
-
-def check_configuration(config, crawler):
-    pass

tests/test_addons/addons.py

@@ -1,33 +0,0 @@
-from zope.interface import implementer
-
-from scrapy.interfaces import IAddon
-
-
-@implementer(IAddon)
-class GoodAddon(object):
-    name = "GoodAddon"
-    version = "1.0"
-
-    def __init__(self, name=None, version=None):
-        if name is not None:
-            self.name = name
-        if version is not None:
-            self.version = version
-
-    def update_addons(self, config, addons):
-        pass
-
-    def update_settings(self, config, settings):
-        pass
-
-    def check_configuration(self, config, crawler):
-        pass
-
-
-@implementer(IAddon)
-class BrokenAddon(object):
-    name = "BrokenAddon"
-    # No version
-
-
-_addon = GoodAddon()

tests/test_crawl.py

@@ -412,7 +412,6 @@ with multiples lines
     def test_abort_on_addon_failed_check(self):
         class FailedCheckAddon(Addon):
             name = "FailedCheckAddon"
-            version = "1.0"
 
             def check_configuration(self, config, crawler):
                 raise ValueError

tests/test_crawler.py

@@ -15,7 +15,6 @@ from twisted.trial import unittest
 from w3lib import __version__ as w3lib_version
 
 import scrapy
-from scrapy.addons import Addon, AddonManager
 from scrapy.crawler import Crawler, CrawlerProcess, CrawlerRunner
 from scrapy.exceptions import ScrapyDeprecationWarning
 from scrapy.extensions import telnet
@@ -56,32 +55,6 @@ class CrawlerTestCase(BaseCrawlerTest):
         self.assertFalse(settings.frozen)
         self.assertTrue(crawler.settings.frozen)
 
-    def test_populate_addons_settings(self):
-        class TestAddon(Addon):
-            name = "TestAddon"
-            version = "1.0"
-
-        addonconfig = {"TEST1": "addon", "TEST2": "addon", "TEST3": "addon"}
-
-        class TestAddon2(Addon):
-            name = "testAddon2"
-            version = "1.0"
-
-        addonconfig2 = {"TEST": "addon2"}
-
-        settings = Settings()
-        settings.set("TESTADDON_TEST1", "project", priority="project")
-        settings.set("TESTADDON_TEST2", "default", priority="default")
-        addonmgr = AddonManager()
-        addonmgr.add(TestAddon(), addonconfig)
-        addonmgr.add(TestAddon2(), addonconfig2)
-        crawler = Crawler(DefaultSpider, settings, addons=addonmgr)
-
-        self.assertEqual(crawler.settings["TESTADDON_TEST1"], "project")
-        self.assertEqual(crawler.settings["TESTADDON_TEST2"], "addon")
-        self.assertEqual(crawler.settings["TESTADDON_TEST3"], "addon")
-        self.assertEqual(crawler.settings["TESTADDON2_TEST"], "addon2")
-
     def test_crawler_accepts_dict(self):
         crawler = get_crawler(DefaultSpider, {"foo": "bar"})
         self.assertEqual(crawler.settings["foo"], "bar")

tests/test_utils_misc/__init__.py

@@ -8,7 +8,6 @@ from scrapy.item import Field, Item
 from scrapy.utils.misc import (
     arg_to_iter,
     create_instance,
-    load_module_or_object,
     load_object,
     rel_has_nofollow,
     set_environ,
@@ -36,12 +35,6 @@ class UtilsMiscTestCase(unittest.TestCase):
         self.assertRaises(NameError, load_object, "scrapy.utils.misc.load_object999")
         self.assertRaises(TypeError, load_object, {})
 
-    def test_load_module_or_object(self):
-        testmod = load_module_or_object(__name__ + ".testmod")
-        self.assertTrue(hasattr(testmod, "TESTVAR"))
-        obj = load_object("scrapy.utils.misc.load_object")
-        self.assertIs(obj, load_object)
-
     def test_walk_modules(self):
         mods = walk_modules("tests.test_utils_misc.test_walk_modules")
         expected = [

tests/test_utils_misc/testmod.py

@@ -1 +0,0 @@
-TESTVAR = True