RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Remove unneeded code.

This commit is contained in:
Andrey Rakhmatullin 2023-06-26 16:57:46 +04:00
parent 4e21e3e59d
commit e7124447f7
16 changed files with 140 additions and 1029 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

241
docs/topics/addons.rst
View File

@ -15,12 +15,13 @@ Activating and configuring add-ons
==================================
Add-ons and their configuration live in Scrapy's
:class:`~scrapy.addons.AddonManager`. During Scrapy's start-up process, and
only then, the add-on manager will read a list of enabled add-ons and their
configurations from your ``ADDONS`` setting.
:class:`~scrapy.addons.AddonManager`. During a :class:`~scrapy.crawler.Crawler`
initialization the add-on manager will read a list of enabled add-ons from your
``ADDONS`` setting and their optional configuration from the respective
settings.
The ``ADDONS`` setting is a dict in which every key is an addon class or its
import path and the vaoue is its priority.
import path and the value is its priority.
The configuration of an add-on, if necessary at all, is stored as a dictionary
setting whose name is the uppercase add-on name.
@ -37,66 +38,17 @@ configuration) are enabled/configured in a project's ``settings.py``::
'some_config': True,
}
Enabling and configuring add-ons within Python code
---------------------------------------------------
The :class:`~scrapy.addons.AddonManager` will only read from Scrapy's settings
*at the beginning* of Scrapy's start-up process.
Afterwards, i.e. as soon as the :class:`~scrapy.addons.AddonManager` is
populated, changing the ``ADDONS`` setting or any of the add-on
configuration dictionary settings will have no effect.
If you want to enable, disable, or configure add-ons in Python code, for example
when writing your own add-on, you will have to use the
:class:`~scrapy.addons.AddonManager`. You can access the add-on manager through
either ``crawler.addons`` or, if you are writing an add-on, through the
``addons`` argument of the :meth:`update_addons` callback. The add-on manager
provides many useful methods and attributes to facilitate interacting with the
add-ons framework, e.g.:
* an :meth:`~scrapy.addons.AddonManager.add` method to load add-ons,
* the :attr:`~scrapy.addons.AddonManager.enabled` list of enabled add-ons,
* :meth:`~scrapy.addons.AddonManager.enable` and
:meth:`~scrapy.addons.AddonManager.disable` methods,
* the :attr:`~scrapy.addons.AddonManager.configs` dictionary which holds the
configuration of all add-ons.
In this example, we ensure that the ``httpcache`` add-on is loaded, and that
its ``expiration_secs`` configuration is set to ``60``::
# addons is an instance of AddonManager
if 'httpcache' not in addons:
addons.add('httpcache', {'expiration_secs': 60})
else:
addons.configs['httpcache']['expiration_secs'] = 60
Writing your own add-ons
========================
Add-ons are (any) Python *objects* that provide Scrapy's *add-on interface*.
The interface is enforced through ``zope.interface``. This leaves the choice of
Python object up the developer. Examples:
* for a small pipeline, the add-on interface could be implemented in the same
class that also implements the ``open/close_spider`` and ``process_item``
callbacks,
* for larger add-ons, or for clearer structure, the interface could be provided
by a stand-alone module.
The absolute minimum interface consists of two attributes:
Add-ons are (any) Python *objects* that provide Scrapy's *add-on interface*:
.. attribute:: name
string with add-on name
.. attribute:: version
version string (PEP-404, e.g. ``'1.0.1'``)
Of course, stating just these two attributes will not get you very far. Add-ons
can provide three callback methods that are called at various stages before the
crawling process:
:type: ``str``
.. method:: update_settings(config, settings)
@ -124,75 +76,18 @@ crawling process:
:param crawler: Fully initialized Scrapy crawler
:type crawler: :class:`~scrapy.crawler.Crawler`
.. method:: update_addons(config, addons)
This method is called immediately before :meth:`update_settings`, and should
be used to enable and configure other *add-ons* only.
Add-ons that are added to the :class:`~scrapy.addons.AddonManager` during
this callback will also have their :meth:`update_addons` method called.
:param config: Configuration of this add-on
:type config: ``dict``
:param addons: Add-on manager holding all loaded add-ons
:type addons: :class:`~scrapy.addons.AddonManager`
Additionally, add-ons may (and should, where appropriate) provide one or more
attributes that can be used for limited automated detection of possible
dependency clashes:
.. attribute:: requires
list of built-in or custom components needed by this add-on, as strings.
.. attribute:: modifies
list of built-in or custom components whose functionality is affected or
replaced by this add-on (a custom HTTP cache should list ``httpcache`` here)
.. attribute:: provides
list of components provided by this add-on (e.g. ``mongodb`` for an
extension that provides generic read/write access to a MongoDB database)
The entries in the :attr:`requires` and :attr:`modifies` attributes can be add-on
names or components from other add-ons' :attr:`provides` attribute. You can
specify :pep:`440`-style information about required versions. Examples::
requires = ['httpcache']
requires = ['otheraddon >= 2.0', 'yetanotheraddon']
The Python object or module that is pointed to by an add-on path (e.g. given in
the ``ADDONS`` setting, or given to
:meth:`~scrapy.addons.AddonManager.add`) does not necessarily have to be an
add-on. Instead, it can provide an ``_addon`` attribute. This attribute can be
either an add-on or another add-on path.
Add-on base class
=================
Scrapy comes with a built-in base class for add-ons which provides some
convenience functionality:
* basic settings can be exported via :meth:`~scrapy.addons.Addon.export_basics`,
configurable via :attr:`~scrapy.addons.Addon.basic_settings`.
* a single component (e.g. an item pipeline or a downloader middleware) can be
inserted into Scrapy's settings via
:meth:`~scrapy.addons.Addon.export_component`, configurable via
:attr:`~scrapy.addons.Addon.component_type`,
:attr:`~scrapy.addons.Addon.component_key`,
:attr:`~scrapy.addons.Addon.component`, and the ``order`` key in
:attr:`~scrapy.addons.Addon.default_config`.
* the add-on configuration can be exposed into Scrapy's settings via
:meth:`~scrapy.addons.Addon.export_config`, configurable via
:attr:`~scrapy.addons.Addon.default_config`,
:attr:`~scrapy.addons.Addon.config_mapping`, and
:attr:`~scrapy.addons.Addon.settings_prefix`.
convenience functionality: the add-on configuration can be exposed into
Scrapy's settings via :meth:`~scrapy.addons.Addon.export_config`, configurable
via :attr:`~scrapy.addons.Addon.default_config` and
:attr:`~scrapy.addons.Addon.config_mapping`.
By default, the base add-on class will expose the add-on configuration into
Scrapy's settings namespace, in caps and with the add-on name prepended. It is
Scrapy's settings namespace, in upper case. It is
easy to write your own functionality while still being able to use the
convenience functions by overwriting
:meth:`~scrapy.addons.Addon.update_settings`.
@ -213,13 +108,11 @@ Set some basic configuration using the :class:`Addon` base class::
class MyAddon(Addon):
name = 'myaddon'
version = '1.0'
component = 'path.to.mypipeline'
component_type = 'ITEM_PIPELINES'
component_order = 200
basic_settings = {
'DNSCACHE_ENABLED': False,
}
def update_settings(self, config, settings):
super().update_settings(settings)
settings["ITEM_PIPELINES"]["path.to.mypipeline"] = 200
settings["DNSCACHE_ENABLED"] = True
Check dependencies::
@ -227,78 +120,22 @@ Check dependencies::
class MyAddon(Addon):
name = 'myaddon'
version = '1.0'
def update_settings(self, config, settings):
try:
import boto
except ImportError:
raise RuntimeError("myaddon requires the boto library")
self.export_config(config, settings)
Enable a component that lives relative to the add-on (see
:ref:`topics-api-settings`)::
from scrapy.addons import Addon
class MyAddon(Addon):
name = 'myaddon'
version = '1.0'
component = __name__ + '.downloadermw.coolmw'
component_type = 'DOWNLOADER_MIDDLEWARES'
component_order = 900
Instantiate components ad hoc::
from path.to.my.pipelines import MySQLPipeline
class MyAddon(object):
name = 'myaddon'
version = '1.0'
def update_settings(self, config, settings):
mysqlpl = MySQLPipeline(password=config['password'])
settings.set(
'ITEM_PIPELINES',
{mysqlpl: 200},
priority='addon',
)
Provide add-on interface along component interface::
class MyPipeline(object):
name = 'mypipeline'
version = '1.0'
def process_item(self, item, spider):
# Do some processing here
return item
def update_settings(self, config, settings):
settings.set(
'ITEM_PIPELINES',
{self: 200},
priority='addon',
)
Enable another addon (see :ref:`topics-api-addonmanager`)::
class MyAddon(object):
name = 'myaddon'
version = '1.0'
def update_addons(self, config, addons):
if 'httpcache' not in addons.enabled:
addons.add('httpcache', {'expiration_secs': 60})
super().update_settings(settings)
Check configuration of fully initialized crawler (see
:ref:`topics-api-crawler`)::
class MyAddon(object):
name = 'myaddon'
version = '1.0'
def update_settings(self, config, settings):
super().update_settings(settings)
settings.set('DNSCACHE_ENABLED', False, priority='addon')
def check_configuration(self, config, crawler):
@ -306,43 +143,3 @@ Check configuration of fully initialized crawler (see
# The spider, some other add-on, or the user messed with the
# DNS cache setting
raise ValueError("myaddon is incompatible with DNS cache")
Provide add-on interface through a module:
.. code-block:: python
name = "AddonModule"
version = "1.0"
class MyPipeline(object):
...
class MyDownloaderMiddleware(object):
...
def update_settings(config, settings):
settings.set(
"ITEM_PIPELINES",
{MyPipeline(): 200},
priority="addon",
)
settings.set(
"DOWNLOADER_MIDDLEWARES",
{MyDownloaderMiddleware(): 800},
priority="addon",
)
Forward to other add-ons depending on Python version::
# This could be a Python module, say project/pipelines/mypipeline.py, but
# could also be done inside a class, etc.
import six
if six.PY3:
# We're running Python 3
_addon = 'path.to.addon'
else:
_addon = 'path.to.other.addon'

347
scrapy/addons.py
View File

@ -1,108 +1,27 @@
import warnings
from collections import OrderedDict, defaultdict
from collections.abc import Mapping
from inspect import isclass
from typing import Dict
from typing import Any, Dict, Iterator, Optional, OrderedDict
import zope.interface
from packaging.requirements import Requirement
from packaging.version import Version
from zope.interface.verify import verifyObject
from scrapy.interfaces import IAddon
from scrapy.utils.conf import build_component_list
from scrapy.utils.misc import load_module_or_object
from scrapy.utils.misc import load_object
@zope.interface.implementer(IAddon)
class Addon(object):
basic_settings = None
"""``dict`` of settings that will be exported via :meth:`export_basics`."""
name: str
default_config = None
"""``dict`` with default configuration."""
config_mapping = None
"""``dict`` with mappings from config names to setting names. The given
setting names will be taken as given, i.e. they will be neither prefixed
nor uppercased.
setting names will be taken as given, not uppercased.
"""
component_type = None
"""Component setting into which to export via :meth:`export_component`. Can
be any of the dictionary-like component setting names. If ``None``,
:meth:`export_component` will do nothing.
"""
component_key = None
"""Key to be used in the component dictionary setting when exporting via
:meth:`export_component`. This is only useful for the settings that have
no order, e.g. ``DOWNLOAD_HANDLERS`` or ``FEED_EXPORTERS``.
"""
component_order = 0
"""Component order to use when not given in the add-on configuration. Has
no effect for component types that use :attr:`component_key`.
"""
component = None
"""Component to be inserted via :meth:`export_component`. This can be
anything that can be used in the dictionary-like component settings, i.e.
a class path or a class. If ``None``, it is assumed that the add-on itself
also provides the component interface, and ``self`` will be used.
"""
settings_prefix = None
"""Prefix with which the add-on configuration will be exported into the
global settings namespace via :meth:`export_config`. If ``None``,
:attr:`name` will be used. If ``False``, no configuration will be exported.
"""
def export_component(self, config, settings):
"""Export the component in :attr:`component` into the dictionary-like
component setting derived from :attr:`component_type`.
Where applicable, the order parameter of the component (i.e. the
dictionary value) will be retrieved from the ``order`` add-on
configuration value.
:param config: Add-on configuration from which to read component order
:type config: ``dict``
:param settings: Settings object into which to export component
:type settings: :class:`~scrapy.settings.Settings`
"""
if self.component_type:
comp = self.component or self
if self.component_key:
# e.g. for DOWNLOAD_HANDLERS: {'http': 'myclass'}
k = self.component_key
v = comp
else:
# e.g. for DOWNLOADER_MIDDLEWARES: {'myclass': 100}
k = comp
v = config.get("order", self.component_order)
settings[self.component_type].update({k: v}, "addon")
def export_basics(self, settings):
"""Export the :attr:`basic_settings` attribute into the settings object.
All settings will be exported with ``addon`` priority (see
:ref:`topics-api-settings`).
:param settings: Settings object into which to expose the basic settings
:type settings: :class:`~scrapy.settings.Settings`
"""
for setting, value in (self.basic_settings or {}).items():
settings.set(setting, value, "addon")
def export_config(self, config, settings):
"""Export the add-on configuration, all keys in caps and with
:attr:`settings_prefix` or :attr:`name` prepended, into the settings
"""Export the add-on configuration, all keys in caps, into the settings
object.
For example, the add-on configuration ``{'key': 'value'}`` will export
the setting ``ADDONNAME_KEY`` with a value of ``value``. All settings
the setting ``KEY`` with a value of ``value``. All settings
will be exported with ``addon`` priority (see
:ref:`topics-api-settings`).
@ -112,11 +31,8 @@ class Addon(object):
:param settings: Settings object into which to export the configuration
:type settings: :class:`~scrapy.settings.Settings`
"""
if self.settings_prefix is False:
return
conf = self.default_config or {}
conf.update(config)
prefix = self.settings_prefix or self.name
# Since default exported config is case-insensitive (everything will be
# uppercased), make mapped config case-insensitive as well
conf_mapping = {k.lower(): v for k, v in (self.config_mapping or {}).items()}
@ -124,14 +40,11 @@ class Addon(object):
if key.lower() in conf_mapping:
key = conf_mapping[key.lower()]
else:
key = (prefix + "_" + key).upper()
key = key.upper()
settings.set(key, val, "addon")
def update_settings(self, config, settings):
"""Export both the basic settings and the add-on configuration. I.e.,
call :meth:`export_basics` and :meth:`export_config`.
For more advanced add-ons, you may want to override this callback.
"""Modifiy `settings` to enable and configure required components.
:param config: Add-on configuration
:type config: ``dict``
@ -139,12 +52,21 @@ class Addon(object):
:param settings: Crawler settings object
:type settings: :class:`~scrapy.settings.Settings`
"""
self.export_component(config, settings)
self.export_basics(settings)
self.export_config(config, settings)
def check_configuration(self, config, crawler):
"""Perform post-initialization checks on fully configured `crawler`.
class AddonManager(Mapping):
:param config: Add-on configuration
:type config: ``dict``
:param crawler: the fully-initialized crawler
:type crawler: :class:`~scrapy.crawler.Crawler`
"""
pass
class AddonManager(Mapping[str, Addon]):
"""This class facilitates loading and storing :ref:`topics-addons`.
You can treat it like a read-only dictionary in which keys correspond to
@ -160,108 +82,42 @@ class AddonManager(Mapping):
"""
def __init__(self):
self._addons = OrderedDict()
self.configs = {}
self._disable_on_add = []
def __init__(self) -> None:
self._addons: OrderedDict[str, Addon] = OrderedDict[str, Addon]()
self.configs: Dict[str, Dict[str, Any]] = {}
def __getitem__(self, name):
def __getitem__(self, name: str) -> Addon:
return self._addons[name]
def __delitem__(self, name):
del self._addons[name]
del self.configs[name]
def __iter__(self):
def __iter__(self) -> Iterator[str]:
return iter(self._addons)
def __len__(self):
def __len__(self) -> int:
return len(self._addons)
def add(self, addon, config=None):
def add(self, addon: Any, config: Optional[Dict[str, Any]] = None):
"""Store an add-on.
If ``addon`` is a string, it will be treated as add-on path and passed
to :meth:`get_addon`. Otherwise, ``addon`` must be a Python object
implementing or providing Scrapy's add-on interface. The interface
will be enforced through ``zope.interface``'s ``verifyObject()``.
If ``addon`` is a class, it will be instantiated. You can avoid this
(for example if you have implemented the add-on callbacks as class
methods) by declaring -- via ``zope.interface`` -- that your class
directly *provides* ``scrapy.interfaces.IAddon``.
:param addon: The add-on object (or path) to be stored
:type addon: Any Python object providing the add-on interface or ``str``
:type addon: Python object, class or ``str``
:param config: The add-on configuration dictionary
:type config: ``dict``
"""
addon = self.get_addon(addon)
if isclass(addon) and not IAddon.providedBy(addon):
if isinstance(addon, (type, str)):
addon = load_object(addon)
if isinstance(addon, type):
addon = addon()
if not IAddon.providedBy(addon):
zope.interface.alsoProvides(addon, IAddon)
# zope.interface's exceptions are already quite helpful. Still, should
# we catch them and log an error message?
verifyObject(IAddon, addon)
name = addon.name
if name in self:
raise ValueError(f"Addon '{name}' already loaded")
self._addons[name] = addon
self.configs[name] = config or {}
if name in self._disable_on_add:
self.configs[name]["_enabled"] = False
self._disable_on_add.remove(name)
def remove(self, addon):
"""Remove an add-on.
If ``addon`` is the name of a stored add-on, that add-on will be
removed. Otherwise, you can use the argument in the same fashion as
in :meth:`add`.
:param addon: The add-on name, object, or path to be removed
:type addon: Any Python object providing the add-on interface or ``str``
"""
if addon in self:
del self[addon]
elif hasattr(addon, "name") and addon.name in self:
del self[addon.name]
else:
try:
del self[self.get_addon(addon).name]
except NameError:
raise KeyError
@staticmethod
def get_addon(path):
"""Get an add-on object by its Python or file path.
``path`` is assumed to be an import path of an add-on. If the object or
module pointed to by ``path`` has an attribute named ``_addon`` that
attribute will be assumed to be the add-on. :meth:`get_addon` will keep
following ``_addon`` attributes until it finds an object that does not
have an attribute named ``_addon``.
:param path: Import path of an add-on
:type path: ``str``
"""
if isinstance(path, str):
try:
obj = load_module_or_object(path)
except NameError:
raise NameError(f"Could not find add-on '{path}'")
else:
obj = path
if hasattr(obj, "_addon"):
obj = AddonManager.get_addon(obj._addon)
return obj
def load_settings(self, settings):
"""Load add-ons and configurations from settings object.
This will invoke :meth:`get_addon` for every add-on path in the
This will load the addon for every add-on path in the
``ADDONS`` setting. For each of these add-ons, the configuration will be
read from the dictionary setting whose name matches the uppercase add-on
name.
@ -271,141 +127,12 @@ class AddonManager(Mapping):
:type settings: :class:`~scrapy.settings.Settings`
"""
paths = build_component_list(settings["ADDONS"])
addons = [self.get_addon(path) for path in paths]
addons = [load_object(path) for path in paths]
configs = [settings.getdict(addon.name.upper()) for addon in addons]
for a, c in zip(addons, configs):
self.add(a, c)
def check_dependency_clashes(self) -> None:
"""Check for incompatibilities in add-on dependencies.
Add-ons can provide information about their dependencies in their
``provides``, ``modifies`` and ``requires`` attributes. This method will
raise an ``ImportError`` if
* a component required by an add-on is not provided by any other add-on,
or
* a component modified by an add-on is not provided by any other add-on,
or
* the same component is provided by more than one add-on,
and warn when a component required by an add-on is modified by any other
add-on.
"""
# Collect all active add-ons and the components they provide
versions: Dict[str, Version] = {}
def add_version(project_name, version):
if project_name in versions:
raise ImportError(
f"Component {project_name} provided by multiple add-ons"
)
versions[project_name] = Version(version)
for name in self:
ver = self[name].version
add_version(name, ver)
for provides_name in getattr(self[name], "provides", []):
add_version(provides_name, ver)
# Collect all required and modified components
def compile_attribute_dict(attribute_name):
attrs = defaultdict(list)
for name in self:
for entry in getattr(self[name], attribute_name, []):
attrs[entry].append(name)
return attrs
modified = compile_attribute_dict("modifies")
required = compile_attribute_dict("requires")
req_or_mod = set(required.keys()).union(modified.keys())
for reqstr in req_or_mod:
req = Requirement(reqstr)
if req.name not in versions or versions[req.name] not in req.specifier:
raise ImportError(
f"Add-ons {required[reqstr] + modified[reqstr]} require"
f" or modify missing component {reqstr}"
)
mod_and_req = set(required.keys()).intersection(modified.keys())
for conflict in mod_and_req:
warnings.warn(
f"Component '{conflict}', required by add-ons {required[conflict]},"
f" is modified by add-ons {modified[conflict]}"
)
def disable(self, addon):
"""Disable an add-on, i.e. prevent its callbacks from being called.
If you disable an add-on before it is loaded, it will be disabled as
soon as it is added to the :class:`AddonManager`.
:param addon: Name of the add-on to be disabled
:type addon: ``str``
"""
if addon in self:
self.configs[addon]["_enabled"] = False
else:
self._disable_on_add.append(addon)
def enable(self, addon):
"""Re-enable a disabled add-on.
Will raise ``ValueError`` if the add-on is neither already loaded nor
marked for being disabled on adding.
:param addon: Name of the add-on to be enabled
:type addon: ``str``
"""
if addon in self:
self.configs[addon]["_enabled"] = True
elif addon in self._disable_on_add:
self._disable_on_add.remove(addon)
else:
raise ValueError("Add-ons need to be added before they can be enabled")
@property
def disabled(self):
"""Names of disabled add-ons"""
return [
a for a in self if not self.configs[a].get("_enabled", True)
] + self._disable_on_add
@property
def enabled(self):
"""Names of enabled add-ons"""
return [a for a in self if self.configs[a].get("_enabled", True)]
def _call_if_exists(self, obj, cbname, *args, **kwargs):
if obj is None:
return
try:
cb = getattr(obj, cbname)
except AttributeError:
return
else:
cb(*args, **kwargs)
def _call_addon(self, addonname, cbname, *args, **kwargs):
if self.configs[addonname].get("_enabled", True):
self._call_if_exists(
self[addonname], cbname, self.configs[addonname], *args, **kwargs
)
def update_addons(self):
"""Call ``update_addons()`` of all held add-ons.
This will also call ``update_addons()`` of all add-ons that are added
last minute during the ``update_addons()`` routine of other add-ons.
"""
called_addons = set()
while called_addons != set(self):
for name in set(self).difference(called_addons):
called_addons.add(name)
self._call_addon(name, "update_addons", self)
def update_settings(self, settings):
def update_settings(self, settings) -> None:
"""Call ``update_settings()`` of all held add-ons.
:param settings: The :class:`~scrapy.settings.Settings` object to be \
@ -413,13 +140,13 @@ class AddonManager(Mapping):
:type settings: :class:`~scrapy.settings.Settings`
"""
for name in self:
self._call_addon(name, "update_settings", settings)
self[name].update_settings(self.configs[name], settings)
def check_configuration(self, crawler):
def check_configuration(self, crawler) -> None:
"""Call ``check_configuration()`` of all held add-ons.
:param crawler: the fully-initialized crawler
:type crawler: :class:`~scrapy.crawler.Crawler`
"""
for name in self:
self._call_addon(name, "check_configuration", crawler)
self[name].check_configuration(self.configs[name], crawler)

5
scrapy/cmdline.py
View File

@ -6,7 +6,6 @@ import sys
from importlib.metadata import entry_points
import scrapy
from scrapy.addons import AddonManager
from scrapy.commands import BaseRunSpiderCommand, ScrapyCommand, ScrapyHelpFormatter
from scrapy.crawler import CrawlerProcess
from scrapy.exceptions import UsageError
@ -131,8 +130,6 @@ def execute(argv=None, settings=None):
else:
settings["EDITOR"] = editor
addons = AddonManager()
inproject = inside_project()
cmds = _get_commands_dict(settings, inproject)
cmdname = _pop_command_name(argv)
@ -156,7 +153,7 @@ def execute(argv=None, settings=None):
opts, args = parser.parse_known_args(args=argv[1:])
_run_print_help(parser, cmd.process_options, args, opts)
cmd.crawler_process = CrawlerProcess(settings, addons=addons)
cmd.crawler_process = CrawlerProcess(settings)
_run_print_help(parser, _run_command, cmd, args, opts)
sys.exit(cmd.exitcode)

23
scrapy/crawler.py
View File

@ -58,7 +58,6 @@ class Crawler:
spidercls: Type[Spider],
settings: Union[None, dict, Settings] = None,
init_reactor: bool = False,
addons: Optional[AddonManager] = None,
):
if isinstance(spidercls, Spider):
raise ValueError("The spidercls argument must be a class, not an object")
@ -70,10 +69,8 @@ class Crawler:
self.settings: Settings = settings.copy()
self.spidercls.update_settings(self.settings)
self.addons: AddonManager = addons if addons is not None else AddonManager()
self.addons: AddonManager = AddonManager()
self.addons.load_settings(self.settings)
self.addons.update_addons()
self.addons.check_dependency_clashes()
self.addons.update_settings(self.settings)
self.signals: SignalManager = SignalManager(self)
@ -200,15 +197,10 @@ class CrawlerRunner:
)
return loader_cls.from_settings(settings.frozencopy())
def __init__(
self,
settings: Union[Dict[str, Any], Settings, None] = None,
addons: Optional[AddonManager] = None,
):
def __init__(self, settings: Union[Dict[str, Any], Settings, None] = None):
if isinstance(settings, dict) or settings is None:
settings = Settings(settings)
self.settings = settings
self.addons: Optional[AddonManager] = addons
self.spider_loader = self._get_spider_loader(settings)
self._crawlers: Set[Crawler] = set()
self._active: Set[defer.Deferred] = set()
@ -292,7 +284,7 @@ class CrawlerRunner:
def _create_crawler(self, spidercls: Union[str, Type[Spider]]) -> Crawler:
if isinstance(spidercls, str):
spidercls = self.spider_loader.load(spidercls)
return Crawler(spidercls, self.settings, addons=self.addons)
return Crawler(spidercls, self.settings)
def stop(self):
"""
@ -338,13 +330,8 @@ class CrawlerProcess(CrawlerRunner):
process. See :ref:`run-from-script` for an example.
"""
def __init__(
self,
settings=None,
install_root_handler: bool = True,
addons: Optional[AddonManager] = None,
):
super().__init__(settings, addons)
def __init__(self, settings=None, install_root_handler=True):
super().__init__(settings)
configure_logging(self.settings, install_root_handler)
log_scrapy_info(self.settings)
self._initialized_reactor = False

21
scrapy/interfaces.py
View File

@ -1,4 +1,4 @@
from zope.interface import Attribute, Interface
from zope.interface import Interface
class ISpiderLoader(Interface):
@ -15,22 +15,3 @@ class ISpiderLoader(Interface):
def find_by_request(request):
"""Return the list of spiders names that can handle the given request"""
class IAddon(Interface):
"""Scrapy add-on"""
name = Attribute("""Add-on name""")
version = Attribute("""Add-on version string (PEP440)""")
# XXX: Can methods be declared optional? I.e., can I enforce the signature
# but not the existence of a method?
# def update_addons(config, addons):
# """Enables and configures other add-ons"""
# def update_settings(config, settings):
# """Modifies `settings` to enable and configure required components"""
# def check_configuration(config, crawler):
# """Performs post-initialization checks on fully configured `crawler`"""

12
scrapy/utils/conf.py
View File

@ -99,18 +99,12 @@ def init_env(project="default", set_syspath=True):
sys.path.append(projdir)
def config_from_filepath(sources):
"""Create a ConfigParser and read in the given `sources`, which can be
either a filename or a list of filenames."""
cfg = ConfigParser()
cfg.read(sources)
return cfg
def get_config(use_closest=True):
"""Get Scrapy config file as a ConfigParser"""
sources = get_sources(use_closest)
return config_from_filepath(sources)
cfg = ConfigParser()
cfg.read(sources)
return cfg
def get_sources(use_closest=True) -> List[str]:

18
scrapy/utils/misc.py
View File

@ -67,7 +67,7 @@ def load_object(path: Union[str, Callable]) -> Any:
if callable(path):
return path
raise TypeError(
"Unexpected argument type, expected string " f"or object, got: {type(path)}"
f"Unexpected argument type, expected string or object, got: {type(path)}"
)
try:
@ -86,22 +86,6 @@ def load_object(path: Union[str, Callable]) -> Any:
return obj
def load_module_or_object(path):
"""Load python module or (non-module) object from given path.
Path can be both a Python or a file path.
"""
try:
return import_module(path)
except ImportError:
pass
try:
return load_object(path)
except (ValueError, NameError, ImportError):
pass
raise NameError(f"Could not load '{path}'")
def walk_modules(path: str) -> List[ModuleType]:
"""Loads a module and all its submodules from the given module path and
returns them. If *any* module throws an exception while importing, that

3
scrapy/utils/test.py
View File

@ -73,7 +73,6 @@ def get_crawler(
spidercls: Optional[Type[Spider]] = None,
settings_dict: Optional[Dict[str, Any]] = None,
prevent_warnings: bool = True,
addons=None,
) -> Crawler:
"""Return an unconfigured Crawler object. If settings_dict is given, it
will be used to populate the crawler settings with a project level
@ -87,7 +86,7 @@ def get_crawler(
if prevent_warnings:
settings["REQUEST_FINGERPRINTER_IMPLEMENTATION"] = "2.7"
settings.update(settings_dict or {})
runner = CrawlerRunner(settings, addons)
runner = CrawlerRunner(settings)
return runner.create_crawler(spidercls or Spider)

72
tests/test_addons.py Normal file
View File

@ -0,0 +1,72 @@
import unittest
from scrapy.addons import Addon, AddonManager
from scrapy.settings import BaseSettings
class GoodAddon(object):
name = "GoodAddon"
def update_settings(self, config, settings):
pass
def check_configuration(self, config, crawler):
pass
class AddonTest(unittest.TestCase):
def setUp(self):
class AddonWithAttributes(Addon):
name = "Test"
self.testaddon = AddonWithAttributes()
def test_export_config(self):
settings = BaseSettings()
self.testaddon.config_mapping = {"MAPPED_key": "MAPPING_WORKED"}
self.testaddon.default_config = {"key": 55, "defaultkey": 100}
self.testaddon.export_config(
{"key": 313, "OTHERKEY": True, "mapped_KEY": 99}, settings
)
self.assertEqual(settings["KEY"], 313)
self.assertEqual(settings["DEFAULTKEY"], 100)
self.assertEqual(settings["OTHERKEY"], True)
self.assertNotIn("MAPPED_key", settings)
self.assertNotIn("MAPPED_KEY", settings)
self.assertEqual(settings["MAPPING_WORKED"], 99)
self.assertEqual(settings.getpriority("KEY"), 15)
def test_update_settings(self):
settings = BaseSettings()
settings.set("KEY1", "default", priority="default")
settings.set("KEY2", "project", priority="project")
addon_config = {"key1": "addon", "key2": "addon", "key3": "addon"}
self.testaddon.update_settings(addon_config, settings)
self.assertEqual(settings["KEY1"], "addon")
self.assertEqual(settings["KEY2"], "project")
self.assertEqual(settings["KEY3"], "addon")
class AddonManagerTest(unittest.TestCase):
def setUp(self):
self.manager = AddonManager()
def test_add(self):
manager = AddonManager()
manager.add("tests.test_addons.GoodAddon")
self.assertCountEqual(manager, ["GoodAddon"])
self.assertIsInstance(manager["GoodAddon"], GoodAddon)
def test_load_settings(self):
settings = BaseSettings()
settings.set(
"ADDONS",
{"tests.test_addons.GoodAddon": 0},
)
settings.set("GOODADDON", {"key": "val2"})
manager = AddonManager()
manager.load_settings(settings)
self.assertCountEqual(manager, ["GoodAddon"])
self.assertIsInstance(manager["GoodAddon"], GoodAddon)
self.assertCountEqual(manager.configs["GoodAddon"], ["key"])
self.assertEqual(manager.configs["GoodAddon"]["key"], "val2")

342
tests/test_addons/__init__.py
View File

@ -1,342 +0,0 @@
import itertools
import unittest
import warnings
from unittest import mock
from zope.interface import directlyProvides
from zope.interface.exceptions import BrokenImplementation, MultipleInvalid
from zope.interface.verify import verifyObject
from scrapy.addons import Addon, AddonManager
from scrapy.crawler import Crawler
from scrapy.interfaces import IAddon
from scrapy.settings import BaseSettings
from . import addonmod, addons
class AddonTest(unittest.TestCase):
def setUp(self):
self.rawaddon = Addon()
class AddonWithAttributes(Addon):
name = "Test"
version = "1.0"
self.testaddon = AddonWithAttributes()
def test_interface(self):
# Raw Addon should fail exactly b/c name and version are not given
self.assertFalse(hasattr(self.rawaddon, "name"))
self.assertFalse(hasattr(self.rawaddon, "version"))
self.assertRaises(MultipleInvalid, verifyObject, IAddon, self.rawaddon)
verifyObject(IAddon, self.testaddon)
def test_export_component(self):
settings = BaseSettings(
{"ITEM_PIPELINES": BaseSettings(), "DOWNLOAD_HANDLERS": BaseSettings()},
"default",
)
self.testaddon.component_type = None
self.testaddon.export_component({}, settings)
self.assertEqual(len(settings["ITEM_PIPELINES"]), 0)
self.testaddon.component_type = "ITEM_PIPELINES"
self.testaddon.component = "test.component"
self.testaddon.export_component({}, settings)
self.assertCountEqual(settings["ITEM_PIPELINES"], ["test.component"])
self.assertEqual(settings["ITEM_PIPELINES"]["test.component"], 0)
self.testaddon.component_order = 313
self.testaddon.export_component({}, settings)
self.assertEqual(settings["ITEM_PIPELINES"]["test.component"], 313)
self.testaddon.component_type = "DOWNLOAD_HANDLERS"
self.testaddon.component_key = "http"
self.testaddon.export_component({}, settings)
self.assertEqual(settings["DOWNLOAD_HANDLERS"]["http"], "test.component")
def test_export_basics(self):
settings = BaseSettings()
self.testaddon.basic_settings = {"TESTKEY": 313, "OTHERKEY": True}
self.testaddon.export_basics(settings)
self.assertEqual(settings["TESTKEY"], 313)
self.assertEqual(settings["OTHERKEY"], True)
self.assertEqual(settings.getpriority("TESTKEY"), 15)
def test_export_config(self):
settings = BaseSettings()
self.testaddon.settings_prefix = None
self.testaddon.config_mapping = {"MAPPED_key": "MAPPING_WORKED"}
self.testaddon.default_config = {"key": 55, "defaultkey": 100}
self.testaddon.export_config(
{"key": 313, "OTHERKEY": True, "mapped_KEY": 99}, settings
)
self.assertEqual(settings["TEST_KEY"], 313)
self.assertEqual(settings["TEST_DEFAULTKEY"], 100)
self.assertEqual(settings["TEST_OTHERKEY"], True)
self.assertNotIn("MAPPED_key", settings)
self.assertNotIn("MAPPED_KEY", settings)
self.assertEqual(settings["MAPPING_WORKED"], 99)
self.assertEqual(settings.getpriority("TEST_KEY"), 15)
self.testaddon.settings_prefix = "PREF"
self.testaddon.export_config({"newkey": 99}, settings)
self.assertEqual(settings["PREF_NEWKEY"], 99)
with mock.patch.object(settings, "set") as mock_set:
self.testaddon.settings_prefix = False
self.testaddon.export_config({"thirdnewkey": 99}, settings)
self.assertEqual(mock_set.call_count, 0)
def test_update_settings(self):
settings = BaseSettings()
settings.set("TEST_KEY1", "default", priority="default")
settings.set("TEST_KEY2", "project", priority="project")
self.testaddon.settings_prefix = None
self.testaddon.basic_settings = {"OTHERTEST_KEY": "addon"}
addon_config = {"key1": "addon", "key2": "addon", "key3": "addon"}
self.testaddon.update_settings(addon_config, settings)
self.assertEqual(settings["OTHERTEST_KEY"], "addon")
self.assertEqual(settings["TEST_KEY1"], "addon")
self.assertEqual(settings["TEST_KEY2"], "project")
self.assertEqual(settings["TEST_KEY3"], "addon")
class AddonManagerTest(unittest.TestCase):
def setUp(self):
self.manager = AddonManager()
def test_add(self):
manager = AddonManager()
manager.add(addonmod, {"key": "val1"})
manager.add("tests.test_addons.addons.GoodAddon")
self.assertCountEqual(manager, ["AddonModule", "GoodAddon"])
self.assertIsInstance(manager["GoodAddon"], addons.GoodAddon)
self.assertCountEqual(manager.configs["AddonModule"], ["key"])
self.assertEqual(manager.configs["AddonModule"]["key"], "val1")
self.assertRaises(ValueError, manager.add, addonmod)
def test_add_dont_instantiate_providing_classes(self):
class ProviderGoodAddon(addons.GoodAddon):
pass
directlyProvides(ProviderGoodAddon, IAddon)
manager = AddonManager()
manager.add(ProviderGoodAddon)
self.assertIs(manager["GoodAddon"], ProviderGoodAddon)
def test_add_verifies(self):
brokenaddon = self.manager.get_addon("tests.test_addons.addons.BrokenAddon")
self.assertRaises(
BrokenImplementation,
self.manager.add,
brokenaddon,
)
def test_add_adds_missing_interface_declaration(self):
class GoodAddonWithoutDeclaration(object):
name = "GoodAddonWithoutDeclaration"
version = "1.0"
self.manager.add(GoodAddonWithoutDeclaration)
def test_remove(self):
manager = AddonManager()
def test_gets_removed(removearg):
manager.add(addonmod)
self.assertIn("AddonModule", manager)
manager.remove(removearg)
self.assertNotIn("AddonModule", manager)
test_gets_removed("AddonModule")
test_gets_removed(addonmod)
test_gets_removed("tests.test_addons.addonmod")
self.assertRaises(KeyError, manager.remove, "nonexistent")
self.assertRaises(KeyError, manager.remove, addons.GoodAddon())
def test_get_addon(self):
goodaddon = self.manager.get_addon("tests.test_addons.addons.GoodAddon")
self.assertIs(goodaddon, addons.GoodAddon)
loaded_addonmod = self.manager.get_addon("tests.test_addons.addonmod")
self.assertIs(loaded_addonmod, addonmod)
goodaddon = self.manager.get_addon("tests.test_addons.addons")
self.assertIsInstance(goodaddon, addons.GoodAddon)
self.assertRaises(NameError, self.manager.get_addon, "xy.n_onexistent")
def test_get_addon_forward(self):
class SomeCls(object):
_addon = "tests.test_addons.addons.GoodAddon"
self.assertIs(self.manager.get_addon(SomeCls()), addons.GoodAddon)
def test_get_addon_nested(self):
x = addons.GoodAddon("outer")
x._addon = addons.GoodAddon("middle")
x._addon._addon = addons.GoodAddon("inner")
self.assertIs(self.manager.get_addon(x), x._addon._addon)
def test_load_settings(self):
settings = BaseSettings()
settings.set(
"ADDONS",
{"tests.test_addons.addonmod": 0, "tests.test_addons.addons.GoodAddon": 0},
)
settings.set("ADDONMODULE", {"key": "val1"})
settings.set("GOODADDON", {"key": "val2"})
manager = AddonManager()
manager.load_settings(settings)
self.assertCountEqual(manager, ["GoodAddon", "AddonModule"])
self.assertIsInstance(manager["GoodAddon"], addons.GoodAddon)
self.assertCountEqual(manager.configs["GoodAddon"], ["key"])
self.assertEqual(manager.configs["GoodAddon"]["key"], "val2")
self.assertEqual(manager["AddonModule"], addonmod)
self.assertIn("key", manager.configs["AddonModule"])
self.assertEqual(manager.configs["AddonModule"]["key"], "val1")
def test_load_settings_order(self):
# Get three addons named 0, 1, 2
addonlist = [addons.GoodAddon(str(x)) for x in range(3)]
# Test for every possible ordering
for ordered_addons in itertools.permutations(addonlist):
expected_order = [a.name for a in ordered_addons]
settings = BaseSettings(
{"ADDONS": {a: i for i, a in enumerate(ordered_addons)}}
)
manager = AddonManager()
manager.load_settings(settings)
self.assertEqual(list(manager.keys()), expected_order)
def test_enabled_disabled(self):
manager = AddonManager()
manager.add(addons.GoodAddon("FirstAddon"))
manager.add(addons.GoodAddon("SecondAddon"))
self.assertEqual(set(manager.enabled), set(("FirstAddon", "SecondAddon")))
self.assertEqual(manager.disabled, [])
manager.disable("FirstAddon")
self.assertEqual(manager.enabled, ["SecondAddon"])
self.assertEqual(manager.disabled, ["FirstAddon"])
manager.enable("FirstAddon")
self.assertEqual(set(manager.enabled), set(("FirstAddon", "SecondAddon")))
self.assertEqual(manager.disabled, [])
def test_enable_before_add(self):
manager = AddonManager()
self.assertRaises(ValueError, manager.enable, "FirstAddon")
manager.disable("FirstAddon")
manager.enable("FirstAddon")
manager.add(addons.GoodAddon("FirstAddon"))
self.assertIn("FirstAddon", manager.enabled)
def test_disable_before_add(self):
manager = AddonManager()
manager.disable("FirstAddon")
manager.add(addons.GoodAddon("FirstAddon"))
self.assertEqual(manager.disabled, ["FirstAddon"])
def test_callbacks(self):
first_addon = addons.GoodAddon("FirstAddon")
second_addon = addons.GoodAddon("SecondAddon")
manager = AddonManager()
manager.add(first_addon, {"test": "first"})
manager.add(second_addon, {"test": "second"})
crawler = mock.create_autospec(Crawler)
settings = BaseSettings()
with mock.patch.object(
first_addon, "update_addons"
) as ua_first, mock.patch.object(
second_addon, "update_addons"
) as ua_second, mock.patch.object(
first_addon, "update_settings"
) as us_first, mock.patch.object(
second_addon, "update_settings"
) as us_second, mock.patch.object(
first_addon, "check_configuration"
) as cc_first, mock.patch.object(
second_addon, "check_configuration"
) as cc_second:
manager.update_addons()
ua_first.assert_called_once_with(manager.configs["FirstAddon"], manager)
ua_second.assert_called_once_with(manager.configs["SecondAddon"], manager)
manager.update_settings(settings)
us_first.assert_called_once_with(manager.configs["FirstAddon"], settings)
us_second.assert_called_once_with(manager.configs["SecondAddon"], settings)
manager.check_configuration(crawler)
cc_first.assert_called_once_with(manager.configs["FirstAddon"], crawler)
cc_second.assert_called_once_with(manager.configs["SecondAddon"], crawler)
self.assertEqual(ua_first.call_count, 1)
self.assertEqual(ua_second.call_count, 1)
self.assertEqual(us_first.call_count, 1)
self.assertEqual(us_second.call_count, 1)
us_first.reset_mock()
us_second.reset_mock()
manager.disable("FirstAddon")
manager.update_settings(settings)
self.assertEqual(us_first.call_count, 0)
manager.enable("FirstAddon")
manager.update_settings(settings)
self.assertEqual(us_first.call_count, 1)
self.assertEqual(us_second.call_count, 2)
# This will become relevant when we let spiders implement the add-on
# interface and should be replaced with a test where
# AddonManager.spidercls = None then.
manager._call_if_exists(None, "irrelevant")
def test_update_addons_last_minute_add(self):
class AddedAddon(addons.GoodAddon):
name = "AddedAddon"
class FirstAddon(addons.GoodAddon):
name = "FirstAddon"
def update_addons(self, config, addons):
addons.add(AddedAddon())
manager = AddonManager()
first_addon = FirstAddon()
with mock.patch.object(
first_addon, "update_addons", wraps=first_addon.update_addons
) as ua_first, mock.patch.object(AddedAddon, "update_addons") as ua_added:
manager.add(first_addon, {"non-empty": "dict"})
manager.update_addons()
self.assertCountEqual(manager, ["FirstAddon", "AddedAddon"])
ua_first.assert_called_once_with(manager.configs["FirstAddon"], manager)
ua_added.assert_called_once_with(manager.configs["AddedAddon"], manager)
def test_check_dependency_clashes_attributes(self):
provides = addons.GoodAddon("ProvidesAddon")
provides.provides = ("test",)
provides2 = addons.GoodAddon("ProvidesAddon2")
provides2.provides = ("test",)
requires = addons.GoodAddon("RequiresAddon")
requires.requires = ("test",)
requires_name = addons.GoodAddon("RequiresNameAddon")
requires_name.requires = ("ProvidesAddon",)
requires_newer = addons.GoodAddon("RequiresNewerAddon")
requires_newer.requires = ("test>=2.0",)
modifies = addons.GoodAddon("ModifiesAddon")
modifies.modifies = ("test",)
def check_with(*addons):
manager = AddonManager()
for a in addons:
manager.add(a)
return manager.check_dependency_clashes()
self.assertRaises(ImportError, check_with, requires)
self.assertRaises(ImportError, check_with, modifies)
self.assertRaises(ImportError, check_with, provides, provides2)
self.assertRaises(ImportError, check_with, provides, requires_newer)
with warnings.catch_warnings(record=True) as w:
check_with(provides, modifies)
check_with(provides)
check_with(provides, requires)
check_with(provides, requires_name)
self.assertEqual(len(w), 0)
check_with(requires, provides, modifies)
self.assertEqual(len(w), 1)

16
tests/test_addons/addonmod.py
View File

@ -1,16 +0,0 @@
from zope.interface import moduleProvides
from scrapy.interfaces import IAddon
moduleProvides(IAddon)
name = "AddonModule"
version = "1.0"
def update_settings(config, settings):
pass
def check_configuration(config, crawler):
pass

33
tests/test_addons/addons.py
View File

@ -1,33 +0,0 @@
from zope.interface import implementer
from scrapy.interfaces import IAddon
@implementer(IAddon)
class GoodAddon(object):
name = "GoodAddon"
version = "1.0"
def __init__(self, name=None, version=None):
if name is not None:
self.name = name
if version is not None:
self.version = version
def update_addons(self, config, addons):
pass
def update_settings(self, config, settings):
pass
def check_configuration(self, config, crawler):
pass
@implementer(IAddon)
class BrokenAddon(object):
name = "BrokenAddon"
# No version
_addon = GoodAddon()

1
tests/test_crawl.py
View File

@ -412,7 +412,6 @@ with multiples lines
def test_abort_on_addon_failed_check(self):
class FailedCheckAddon(Addon):
name = "FailedCheckAddon"
version = "1.0"
def check_configuration(self, config, crawler):
raise ValueError

27
tests/test_crawler.py
View File

@ -15,7 +15,6 @@ from twisted.trial import unittest
from w3lib import __version__ as w3lib_version
import scrapy
from scrapy.addons import Addon, AddonManager
from scrapy.crawler import Crawler, CrawlerProcess, CrawlerRunner
from scrapy.exceptions import ScrapyDeprecationWarning
from scrapy.extensions import telnet
@ -56,32 +55,6 @@ class CrawlerTestCase(BaseCrawlerTest):
self.assertFalse(settings.frozen)
self.assertTrue(crawler.settings.frozen)
def test_populate_addons_settings(self):
class TestAddon(Addon):
name = "TestAddon"
version = "1.0"
addonconfig = {"TEST1": "addon", "TEST2": "addon", "TEST3": "addon"}
class TestAddon2(Addon):
name = "testAddon2"
version = "1.0"
addonconfig2 = {"TEST": "addon2"}
settings = Settings()
settings.set("TESTADDON_TEST1", "project", priority="project")
settings.set("TESTADDON_TEST2", "default", priority="default")
addonmgr = AddonManager()
addonmgr.add(TestAddon(), addonconfig)
addonmgr.add(TestAddon2(), addonconfig2)
crawler = Crawler(DefaultSpider, settings, addons=addonmgr)
self.assertEqual(crawler.settings["TESTADDON_TEST1"], "project")
self.assertEqual(crawler.settings["TESTADDON_TEST2"], "addon")
self.assertEqual(crawler.settings["TESTADDON_TEST3"], "addon")
self.assertEqual(crawler.settings["TESTADDON2_TEST"], "addon2")
def test_crawler_accepts_dict(self):
crawler = get_crawler(DefaultSpider, {"foo": "bar"})
self.assertEqual(crawler.settings["foo"], "bar")

7
tests/test_utils_misc/__init__.py
View File

@ -8,7 +8,6 @@ from scrapy.item import Field, Item
from scrapy.utils.misc import (
arg_to_iter,
create_instance,
load_module_or_object,
load_object,
rel_has_nofollow,
set_environ,
@ -36,12 +35,6 @@ class UtilsMiscTestCase(unittest.TestCase):
self.assertRaises(NameError, load_object, "scrapy.utils.misc.load_object999")
self.assertRaises(TypeError, load_object, {})
def test_load_module_or_object(self):
testmod = load_module_or_object(__name__ + ".testmod")
self.assertTrue(hasattr(testmod, "TESTVAR"))
obj = load_object("scrapy.utils.misc.load_object")
self.assertIs(obj, load_object)
def test_walk_modules(self):
mods = walk_modules("tests.test_utils_misc.test_walk_modules")
expected = [

1
tests/test_utils_misc/testmod.py
View File

@ -1 +0,0 @@
TESTVAR = True