RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #1586 from jdemaeyer/fix/backwards-compatible-per-key-priorities 

[MRG+1] Backwards compatible per key priorities
This commit is contained in:
Julia Medina 2015-11-11 14:08:50 -03:00
parent 9548691fdd 4f364764ae
commit 54216d7afe
20 changed files with 269 additions and 186 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

22
docs/topics/downloader-middleware.rst
View File

@ -23,20 +23,22 @@ Here's an example::
'myproject.middlewares.CustomDownloaderMiddleware': 543,
}
The specified :setting:`DOWNLOADER_MIDDLEWARES` setting is merged with the
default one (i.e. it does not overwrite it) and then sorted by order to get the
final sorted list of enabled middlewares: the first middleware is the one
closer to the engine and the last is the one closer to the downloader.
The :setting:`DOWNLOADER_MIDDLEWARES` setting is merged with the
:setting:`DOWNLOADER_MIDDLEWARES_BASE` setting defined in Scrapy (and not meant
to be overridden) and then sorted by order to get the final sorted list of
enabled middlewares: the first middleware is the one closer to the engine and
the last is the one closer to the downloader.
To decide which order to assign to your middleware see the default
:setting:`DOWNLOADER_MIDDLEWARES` setting and pick a value according to
To decide which order to assign to your middleware see the
:setting:`DOWNLOADER_MIDDLEWARES_BASE` setting and pick a value according to
where you want to insert the middleware. The order does matter because each
middleware performs a different action and your middleware could depend on some
previous (or subsequent) middleware being applied.
If you want to disable a built-in middleware you must define it in your
project's :setting:`DOWNLOADER_MIDDLEWARES` setting and assign ``None`` as its
value. For example, if you want to disable the user-agent middleware::
If you want to disable a built-in middleware (the ones defined in
:setting:`DOWNLOADER_MIDDLEWARES_BASE` and enabled by default) you must define it
in your project's :setting:`DOWNLOADER_MIDDLEWARES` setting and assign `None`
as its value. For example, if you want to disable the user-agent middleware::
DOWNLOADER_MIDDLEWARES = {
'myproject.middlewares.CustomDownloaderMiddleware': 543,
@ -162,7 +164,7 @@ middleware, see the :ref:`downloader middleware usage guide
<topics-downloader-middleware>`.
For a list of the components enabled by default (and their orders) see the
:setting:`DOWNLOADER_MIDDLEWARES` setting.
:setting:`DOWNLOADER_MIDDLEWARES_BASE` setting.
.. _cookies-mw:

13
docs/topics/extensions.rst
View File

@ -42,13 +42,14 @@ by a string: the full Python path to the extension's class name. For example::
As you can see, the :setting:`EXTENSIONS` setting is a dict where the keys are
the extension paths, and their values are the orders, which define the
extension *loading* order. The specified :setting:`EXTENSIONS` setting is merged
with the default one (i.e. it does not overwrite it) and then sorted by order
to get the final sorted list of enabled extensions.
extension *loading* order. The :setting:`EXTENSIONS` setting is merged with the
:setting:`EXTENSIONS_BASE` setting defined in Scrapy (and not meant to be
overridden) and then sorted by order to get the final sorted list of enabled
extensions.
As extensions typically do not depend on each other, their loading order is
irrelevant in most cases. This is why the default :setting:`EXTENSIONS` setting
defines all extensions with the same order (``500``). However, this feature can
irrelevant in most cases. This is why the :setting:`EXTENSIONS_BASE` setting
defines all extensions with the same order (``0``). However, this feature can
be exploited if you need to add an extension which depends on other extensions
already loaded.
@ -63,7 +64,7 @@ Disabling an extension
======================
In order to disable an extension that comes enabled by default (ie. those
included in the default :setting:`EXTENSIONS` setting) you must set its order to
included in the :setting:`EXTENSIONS_BASE` setting) you must set its order to
``None``. For example::
EXTENSIONS = {

47
docs/topics/feed-exports.rst
View File

@ -265,6 +265,16 @@ Whether to export empty feeds (ie. feeds with no items).
FEED_STORAGES
-------------
Default:: ``{}``
A dict containing additional feed storage backends supported by your project.
The keys are URI schemes and the values are paths to storage classes.
.. setting:: FEED_STORAGES_BASE
FEED_STORAGES_BASE
------------------
Default::
{
@ -275,19 +285,30 @@ Default::
'ftp': 'scrapy.extensions.feedexport.FTPFeedStorage',
}
A dict containing all feed storage backends supported by your project. The keys
are URI schemes and the values are paths to storage classes.
A dict containing the built-in feed storage backends supported by Scrapy. You
can disable any of these backends by assigning ``None`` to their URI scheme in
:setting:`FEED_STORAGES`. E.g., to disable the built-in FTP storage backend
(without replacement), place this in your ``settings.py``::
When you set :setting:`FEED_STORAGES` manually, e.g. in your project's settings
module, it will be merged with the default, not overwrite it. If you want to
disable any of the default feed storage backends, you must assign ``None`` as
their value.
FEED_STORAGES = {
'ftp': None,
}
.. setting:: FEED_EXPORTERS
FEED_EXPORTERS
--------------
Default:: ``{}``
A dict containing additional exporters supported by your project. The keys are
serialization formats and the values are paths to :ref:`Item exporter
<topics-exporters>` classes.
.. setting:: FEED_EXPORTERS_BASE
FEED_EXPORTERS_BASE
-------------------
Default::
{
@ -300,14 +321,14 @@ Default::
'pickle': 'scrapy.exporters.PickleItemExporter',
}
A dict containing all feed exporters supported by your project. The keys are
URI schemes and the values are paths to :ref:`Item exporter <topics-exporters>`
classes.
A dict containing the built-in feed exporters supported by Scrapy. You can
disable any of these exporters by assigning ``None`` to their serialization
format in :setting:`FEED_EXPORTERS`. E.g., to disable the built-in CSV exporter
(without replacement), place this in your ``settings.py``::
When you set :setting:`FEED_EXPORTERS` manually, e.g. in your project's settings
module, it will be merged with the default, not overwrite it. If you want to
disable any of the default feed exporters, you must assign ``None`` as their
value.
FEED_EXPORTERS = {
'csv': None,
}
.. _URI: http://en.wikipedia.org/wiki/Uniform_Resource_Identifier
.. _Amazon S3: http://aws.amazon.com/s3/

128
docs/topics/settings.rst
View File

@ -269,11 +269,6 @@ Default::
The default headers used for Scrapy HTTP Requests. They're populated in the
:class:`~scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware`.
When you set :setting:`DEFAULT_REQUEST_HEADERS` manually, e.g. in your
project's settings module, it will be merged with the default, not overwrite it.
If you want to disable any of the default request headers (and not replace them)
you must assign ``None`` as their value.
.. setting:: DEPTH_LIMIT
DEPTH_LIMIT
@ -355,6 +350,16 @@ The downloader to use for crawling.
DOWNLOADER_MIDDLEWARES
----------------------
Default:: ``{}``
A dict containing the downloader middlewares enabled in your project, and their
orders. For more info see :ref:`topics-downloader-middleware-setting`.
.. setting:: DOWNLOADER_MIDDLEWARES_BASE
DOWNLOADER_MIDDLEWARES_BASE
---------------------------
Default::
{
@ -375,16 +380,11 @@ Default::
'scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware': 900,
}
A dict containing the downloader middlewares enabled in your project, and their
orders. Low orders are closer to the engine, high orders are closer to the
downloader.
When you set :setting:`DOWNLOADER_MIDDLEWARES` manually, e.g. in your project's
settings module, it will be merged with the default, not overwrite it. If you
want to disable any of the default downloader middlewares you must assign
``None`` as their value.
For more info see :ref:`topics-downloader-middleware-setting`.
A dict containing the downloader middlewares enabled by default in Scrapy. Low
orders are closer to the engine, high orders are closer to the downloader. You
should never modify this setting in your project, modify
:setting:`DOWNLOADER_MIDDLEWARES` instead. For more info see
:ref:`topics-downloader-middleware-setting`.
.. setting:: DOWNLOADER_STATS
@ -425,6 +425,16 @@ spider attribute.
DOWNLOAD_HANDLERS
-----------------
Default: ``{}``
A dict containing the request downloader handlers enabled in your project.
See :setting:`DOWNLOAD_HANDLERS_BASE` for example format.
.. setting:: DOWNLOAD_HANDLERS_BASE
DOWNLOAD_HANDLERS_BASE
----------------------
Default::
{
@ -436,15 +446,16 @@ Default::
}
A dict containing the request downloader handlers enabled in your project.
A dict containing the request download handlers enabled by default in Scrapy.
You should never modify this setting in your project, modify
:setting:`DOWNLOAD_HANDLERS` instead.
When you set :setting:`DOWNLOAD_HANDLERS` manually, e.g. in your project's
settings module, it will be merged with the default, not overwrite it. If you
want to disable any of the default download handlers you must assign ``None``
as their value. For example, if you want to disable the file download handler::
You can disable any of these download handlers by assigning ``None`` to their
URI scheme in :setting:`DOWNLOAD_HANDLERS`. E.g., to disable the built-in FTP
handler (without replacement), place this in your ``settings.py``::
DOWNLOAD_HANDLERS = {
'file': None,
'ftp': None,
}
.. setting:: DOWNLOAD_TIMEOUT
@ -544,6 +555,15 @@ to ``vi`` (on Unix systems) or the IDLE editor (on Windows).
EXTENSIONS
----------
Default:: ``{}``
A dict containing the extensions enabled in your project, and their orders.
.. setting:: EXTENSIONS_BASE
EXTENSIONS_BASE
---------------
Default::
{
@ -558,15 +578,10 @@ Default::
'scrapy.extensions.throttle.AutoThrottle': 0,
}
A dict containing the extensions enabled in your project, and their orders. By
default, this setting contains all stable built-in extensions. Keep in mind that
A dict containing the extensions available by default in Scrapy, and their
orders. This setting contains all stable built-in extensions. Keep in mind that
some of them need to be enabled through a setting.
When you set :setting:`EXTENSIONS` manually, e.g. in your project's settings
module, it will be merged with the default, not overwrite it. If you want to
disable any of the default enabled extensions you must assign ``None`` as their
value.
For more information See the :ref:`extensions user guide <topics-extensions>`
and the :ref:`list of available extensions <topics-extensions-ref>`.
@ -589,6 +604,16 @@ Example::
'mybot.pipelines.validate.StoreMyItem': 800,
}
.. setting:: ITEM_PIPELINES_BASE
ITEM_PIPELINES_BASE
-------------------
Default: ``{}``
A dict containing the pipelines enabled by default in Scrapy. You should never
modify this setting in your project, modify :setting:`ITEM_PIPELINES` instead.
.. setting:: LOG_ENABLED
LOG_ENABLED
@ -878,6 +903,16 @@ The scheduler to use for crawling.
SPIDER_CONTRACTS
----------------
Default:: ``{}``
A dict containing the spider contracts enabled in your project, used for
testing spiders. For more info see :ref:`topics-contracts`.
.. setting:: SPIDER_CONTRACTS_BASE
SPIDER_CONTRACTS_BASE
---------------------
Default::
{
@ -886,13 +921,17 @@ Default::
'scrapy.contracts.default.ScrapesContract': 3,
}
A dict containing the scrapy contracts enabled in your project, used for
testing spiders. For more info see :ref:`topics-contracts`.
A dict containing the scrapy contracts enabled by default in Scrapy. You should
never modify this setting in your project, modify :setting:`SPIDER_CONTRACTS`
instead. For more info see :ref:`topics-contracts`.
When you set :setting:`SPIDER_CONTRACTS` manually, e.g. in your project's
settings module, it will be merged with the default, not overwrite it. If you
want to disable any of the default contracts you must assign ``None`` as their
value.
You can disable any of these contracts by assigning ``None`` to their class
path in :setting:`SPIDER_CONTRACTS`. E.g., to disable the built-in
``ScrapesContract``, place this in your ``settings.py``::
SPIDER_CONTRACTS = {
'scrapy.contracts.default.ScrapesContract': None,
}
.. setting:: SPIDER_LOADER_CLASS
@ -909,6 +948,16 @@ The class that will be used for loading spiders, which must implement the
SPIDER_MIDDLEWARES
------------------
Default:: ``{}``
A dict containing the spider middlewares enabled in your project, and their
orders. For more info see :ref:`topics-spider-middleware-setting`.
.. setting:: SPIDER_MIDDLEWARES_BASE
SPIDER_MIDDLEWARES_BASE
-----------------------
Default::
{
@ -919,14 +968,9 @@ Default::
'scrapy.spidermiddlewares.depth.DepthMiddleware': 900,
}
A dict containing the spider middlewares enabled in your project, and their
orders. Low orders are closer to the engine, high orders are closer to the
spider. For more info see :ref:`topics-spider-middleware-setting`.
When you set :setting:`SPIDER_MIDDLEWARES` manually, e.g. in your project's
settings module, it will be merged with the default, not overwrite it. If you
want to disable any of the default spider middlewares you must assign ``None``
as their value.
A dict containing the spider middlewares enabled by default in Scrapy, and
their orders. Low orders are closer to the engine, high orders are closer to
the spider. For more info see :ref:`topics-spider-middleware-setting`.
.. setting:: SPIDER_MODULES

22
docs/topics/spider-middleware.rst
View File

@ -24,20 +24,22 @@ Here's an example::
'myproject.middlewares.CustomSpiderMiddleware': 543,
}
The specified :setting:`SPIDER_MIDDLEWARES` setting is merged with the default
one (i.e. it does not overwrite it) and then sorted by order to get the final
sorted list of enabled middlewares: the first middleware is the one closer to
the engine and the last is the one closer to the spider.
The :setting:`SPIDER_MIDDLEWARES` setting is merged with the
:setting:`SPIDER_MIDDLEWARES_BASE` setting defined in Scrapy (and not meant to
be overridden) and then sorted by order to get the final sorted list of enabled
middlewares: the first middleware is the one closer to the engine and the last
is the one closer to the spider.
To decide which order to assign to your middleware see the default
:setting:`SPIDER_MIDDLEWARES` setting and pick a value according to where
To decide which order to assign to your middleware see the
:setting:`SPIDER_MIDDLEWARES_BASE` setting and pick a value according to where
you want to insert the middleware. The order does matter because each
middleware performs a different action and your middleware could depend on some
previous (or subsequent) middleware being applied.
If you want to disable a builtin middleware you must define it in your project's
:setting:`SPIDER_MIDDLEWARES` setting and assign ``None`` as its value. For
example, if you want to disable the off-site middleware::
If you want to disable a builtin middleware (the ones defined in
:setting:`SPIDER_MIDDLEWARES_BASE`, and enabled by default) you must define it
in your project :setting:`SPIDER_MIDDLEWARES` setting and assign `None` as its
value. For example, if you want to disable the off-site middleware::
SPIDER_MIDDLEWARES = {
'myproject.middlewares.CustomSpiderMiddleware': 543,
@ -171,7 +173,7 @@ information on how to use them and how to write your own spider middleware, see
the :ref:`spider middleware usage guide <topics-spider-middleware>`.
For a list of the components enabled by default (and their orders) see the
:setting:`SPIDER_MIDDLEWARES` setting.
:setting:`SPIDER_MIDDLEWARES_BASE` setting.
DepthMiddleware
---------------

2
scrapy/commands/check.py
View File

@ -58,7 +58,7 @@ class Command(ScrapyCommand):
def run(self, args, opts):
# load contracts
contracts = build_component_list(self.settings._getcomposite('SPIDER_CONTRACTS'))
contracts = build_component_list(self.settings.getwithbase('SPIDER_CONTRACTS'))
conman = ContractsManager(load_object(c) for c in contracts)
runner = TextTestRunner(verbosity=2 if opts.verbose else 1)
result = TextTestResult(runner.stream, runner.descriptions, runner.verbosity)

3
scrapy/commands/crawl.py
View File

@ -35,7 +35,8 @@ class Command(ScrapyCommand):
self.settings.set('FEED_URI', 'stdout:', priority='cmdline')
else:
self.settings.set('FEED_URI', opts.output, priority='cmdline')
feed_exporters = without_none_values(self.settings._getcomposite('FEED_EXPORTERS'))
feed_exporters = without_none_values(
self.settings.getwithbase('FEED_EXPORTERS'))
valid_output_formats = feed_exporters.keys()
if not opts.output_format:
opts.output_format = os.path.splitext(opts.output)[1].replace(".", "")

2
scrapy/commands/runspider.py
View File

@ -58,7 +58,7 @@ class Command(ScrapyCommand):
self.settings.set('FEED_URI', 'stdout:', priority='cmdline')
else:
self.settings.set('FEED_URI', opts.output, priority='cmdline')
feed_exporters = without_none_values(self.settings._getcomposite('FEED_EXPORTERS'))
feed_exporters = without_none_values(self.settings.getwithbase('FEED_EXPORTERS'))
valid_output_formats = feed_exporters.keys()
if not opts.output_format:
opts.output_format = os.path.splitext(opts.output)[1].replace(".", "")

3
scrapy/core/downloader/handlers/__init__.py
View File

@ -20,7 +20,8 @@ class DownloadHandlers(object):
self._schemes = {} # stores acceptable schemes on instancing
self._handlers = {} # stores instanced handlers for schemes
self._notconfigured = {} # remembers failed handlers
handlers = without_none_values(crawler.settings._getcomposite('DOWNLOAD_HANDLERS'))
handlers = without_none_values(
crawler.settings.getwithbase('DOWNLOAD_HANDLERS'))
for scheme, clspath in six.iteritems(handlers):
self._schemes[scheme] = clspath

3
scrapy/core/downloader/middleware.py
View File

@ -19,7 +19,8 @@ class DownloaderMiddlewareManager(MiddlewareManager):
@classmethod
def _get_mwlist_from_settings(cls, settings):
return build_component_list(settings._getcomposite('DOWNLOADER_MIDDLEWARES'))
return build_component_list(
settings.getwithbase('DOWNLOADER_MIDDLEWARES'))
def _add_middleware(self, mw):
if hasattr(mw, 'process_request'):

2
scrapy/core/spidermw.py
View File

@ -18,7 +18,7 @@ class SpiderMiddlewareManager(MiddlewareManager):
@classmethod
def _get_mwlist_from_settings(cls, settings):
return build_component_list(settings._getcomposite('SPIDER_MIDDLEWARES'))
return build_component_list(settings.getwithbase('SPIDER_MIDDLEWARES'))
def _add_middleware(self, mw):
super(SpiderMiddlewareManager, self)._add_middleware(mw)

2
scrapy/extension.py
View File

@ -12,4 +12,4 @@ class ExtensionManager(MiddlewareManager):
@classmethod
def _get_mwlist_from_settings(cls, settings):
return build_component_list(settings._getcomposite('EXTENSIONS'))
return build_component_list(settings.getwithbase('EXTENSIONS'))

2
scrapy/extensions/feedexport.py
View File

@ -196,7 +196,7 @@ class FeedExporter(object):
return item
def _load_components(self, setting_prefix):
conf = without_none_values(self.settings._getcomposite(setting_prefix))
conf = without_none_values(self.settings.getwithbase(setting_prefix))
d = {}
for k, v in conf.items():
try:

2
scrapy/pipelines/__init__.py
View File

@ -13,7 +13,7 @@ class ItemPipelineManager(MiddlewareManager):
@classmethod
def _get_mwlist_from_settings(cls, settings):
return build_component_list(settings._getcomposite('ITEM_PIPELINES'))
return build_component_list(settings.getwithbase('ITEM_PIPELINES'))
def _add_middleware(self, pipe):
super(ItemPipelineManager, self)._add_middleware(pipe)

57
scrapy/settings/__init__.py
View File

@ -49,14 +49,11 @@ class SettingsAttribute(object):
def set(self, value, priority):
"""Sets value if priority is higher or equal than current priority."""
if isinstance(self.value, BaseSettings):
# Ignore self.priority if self.value has per-key priorities
self.value.update(value, priority)
self.priority = max(self.value.maxpriority(), priority)
else:
if priority >= self.priority:
self.value = value
self.priority = priority
if priority >= self.priority:
if isinstance(self.value, BaseSettings):
value = BaseSettings(value, priority=priority)
self.value = value
self.priority = priority
def __str__(self):
return "<SettingsAttribute value={self.value!r} " \
@ -93,10 +90,9 @@ class BaseSettings(MutableMapping):
self.update(values, priority)
def __getitem__(self, opt_name):
value = None
if opt_name in self:
value = self.attributes[opt_name].value
return value
if opt_name not in self:
return None
return self.attributes[opt_name].value
def __contains__(self, name):
return name in self.attributes
@ -195,25 +191,17 @@ class BaseSettings(MutableMapping):
value = json.loads(value)
return dict(value)
def _getcomposite(self, name):
# DO NOT USE THIS FUNCTION IN YOUR CUSTOM PROJECTS
# It's for internal use in the transition away from the _BASE settings
# and will be removed along with _BASE support in a future release
basename = name + "_BASE"
if basename in self:
warnings.warn('_BASE settings are deprecated.',
category=ScrapyDeprecationWarning)
# When users defined a _BASE setting, they explicitly don't want to
# use any of Scrapy's defaults. Therefore, we only use these entries
# from self[name] (where the defaults now live) that have a priority
# higher than 'default'
compsett = BaseSettings(self[basename], priority='default')
for k in self[name]:
prio = self[name].getpriority(k)
if prio > get_settings_priority('default'):
compsett.set(k, self[name][k], prio)
return compsett
return self[name]
def getwithbase(self, name):
"""Get a composition of a dictionary-like setting and its `_BASE`
counterpart.
:param name: name of the dictionary-like setting
:type name: string
"""
compbs = BaseSettings()
compbs.update(self[name + '_BASE'])
compbs.update(self[name])
return compbs
def getpriority(self, name):
"""
@ -223,10 +211,9 @@ class BaseSettings(MutableMapping):
:param name: the setting name
:type name: string
"""
prio = None
if name in self:
prio = self.attributes[name].priority
return prio
if name not in self:
return None
return self.attributes[name].priority
def maxpriority(self):
"""

25
scrapy/settings/default_settings.py
View File

@ -63,7 +63,8 @@ DNS_TIMEOUT = 60
DOWNLOAD_DELAY = 0
DOWNLOAD_HANDLERS = {
DOWNLOAD_HANDLERS = {}
DOWNLOAD_HANDLERS_BASE = {
'file': 'scrapy.core.downloader.handlers.file.FileDownloadHandler',
'http': 'scrapy.core.downloader.handlers.http.HTTPDownloadHandler',
'https': 'scrapy.core.downloader.handlers.http.HTTPDownloadHandler',
@ -81,7 +82,9 @@ DOWNLOADER = 'scrapy.core.downloader.Downloader'
DOWNLOADER_HTTPCLIENTFACTORY = 'scrapy.core.downloader.webclient.ScrapyHTTPClientFactory'
DOWNLOADER_CLIENTCONTEXTFACTORY = 'scrapy.core.downloader.contextfactory.ScrapyClientContextFactory'
DOWNLOADER_MIDDLEWARES = {
DOWNLOADER_MIDDLEWARES = {}
DOWNLOADER_MIDDLEWARES_BASE = {
# Engine side
'scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware': 100,
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware': 300,
@ -113,7 +116,9 @@ except KeyError:
else:
EDITOR = 'vi'
EXTENSIONS = {
EXTENSIONS = {}
EXTENSIONS_BASE = {
'scrapy.extensions.corestats.CoreStats': 0,
'scrapy.extensions.telnet.TelnetConsole': 0,
'scrapy.extensions.memusage.MemoryUsage': 0,
@ -130,14 +135,16 @@ FEED_URI_PARAMS = None # a function to extend uri arguments
FEED_FORMAT = 'jsonlines'
FEED_STORE_EMPTY = False
FEED_EXPORT_FIELDS = None
FEED_STORAGES = {
FEED_STORAGES = {}
FEED_STORAGES_BASE = {
'': 'scrapy.extensions.feedexport.FileFeedStorage',
'file': 'scrapy.extensions.feedexport.FileFeedStorage',
'stdout': 'scrapy.extensions.feedexport.StdoutFeedStorage',
's3': 'scrapy.extensions.feedexport.S3FeedStorage',
'ftp': 'scrapy.extensions.feedexport.FTPFeedStorage',
}
FEED_EXPORTERS = {
FEED_EXPORTERS = {}
FEED_EXPORTERS_BASE = {
'json': 'scrapy.exporters.JsonItemExporter',
'jsonlines': 'scrapy.exporters.JsonLinesItemExporter',
'jl': 'scrapy.exporters.JsonLinesItemExporter',
@ -163,6 +170,7 @@ HTTPCACHE_GZIP = False
ITEM_PROCESSOR = 'scrapy.pipelines.ItemPipelineManager'
ITEM_PIPELINES = {}
ITEM_PIPELINES_BASE = {}
LOG_ENABLED = True
LOG_ENCODING = 'utf-8'
@ -221,7 +229,9 @@ SCHEDULER_MEMORY_QUEUE = 'scrapy.squeues.LifoMemoryQueue'
SPIDER_LOADER_CLASS = 'scrapy.spiderloader.SpiderLoader'
SPIDER_MIDDLEWARES = {
SPIDER_MIDDLEWARES = {}
SPIDER_MIDDLEWARES_BASE = {
# Engine side
'scrapy.spidermiddlewares.httperror.HttpErrorMiddleware': 50,
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware': 500,
@ -248,7 +258,8 @@ TELNETCONSOLE_ENABLED = 1
TELNETCONSOLE_PORT = [6023, 6073]
TELNETCONSOLE_HOST = '127.0.0.1'
SPIDER_CONTRACTS = {
SPIDER_CONTRACTS = {}
SPIDER_CONTRACTS_BASE = {
'scrapy.contracts.default.UrlContract': 1,
'scrapy.contracts.default.ReturnsContract': 2,
'scrapy.contracts.default.ScrapesContract': 3,

14
scrapy/utils/conf.py
View File

@ -10,7 +10,7 @@ from scrapy.utils.deprecate import update_classpath
from scrapy.utils.python import without_none_values
def build_component_list(compdict, convert=update_classpath):
def build_component_list(compdict, custom=None, convert=update_classpath):
"""Compose a component list from a { class: order } dictionary."""
def _check_components(complist):
@ -34,9 +34,15 @@ def build_component_list(compdict, convert=update_classpath):
_check_components(compdict)
return {convert(k): v for k, v in six.iteritems(compdict)}
if isinstance(compdict, (list, tuple)):
_check_components(compdict)
return type(compdict)(convert(c) for c in compdict)
# BEGIN Backwards compatibility for old (base, custom) call signature
if isinstance(custom, (list, tuple)):
_check_components(custom)
return type(custom)(convert(c) for c in custom)
if custom is not None:
compdict.update(custom)
# END Backwards compatibility
compdict = without_none_values(_map_keys(compdict))
return [k for k, v in sorted(six.iteritems(compdict), key=itemgetter(1))]

19
tests/test_cmdline/__init__.py
View File

@ -1,10 +1,11 @@
import os
import json
import sys
import shutil
import os
import pstats
import tempfile
import shutil
import six
from subprocess import Popen, PIPE
import sys
import tempfile
import unittest
try:
from cStringIO import StringIO
@ -57,14 +58,14 @@ class CmdlineTest(unittest.TestCase):
shutil.rmtree(path)
def test_override_dict_settings(self):
EXT_PATH = "tests.test_cmdline.extensions.DummyExtension"
EXTENSIONS = {EXT_PATH: 200}
settingsstr = self._execute('settings', '--get', 'EXTENSIONS', '-s',
('EXTENSIONS={"tests.test_cmdline.extensions.TestExtension": '
'100, "tests.test_cmdline.extensions.DummyExtension": 200}'))
'EXTENSIONS=' + json.dumps(EXTENSIONS))
# XXX: There's gotta be a smarter way to do this...
self.assertNotIn("...", settingsstr)
for char in ("'", "<", ">", 'u"'):
settingsstr = settingsstr.replace(char, '"')
settingsdict = json.loads(settingsstr)
self.assertIn('tests.test_cmdline.extensions.DummyExtension', settingsdict)
self.assertIn('value=200', settingsdict['tests.test_cmdline.extensions.DummyExtension'])
self.assertIn('value=100', settingsdict['tests.test_cmdline.extensions.TestExtension'])
six.assertCountEqual(self, settingsdict.keys(), EXTENSIONS.keys())
self.assertIn('value=200', settingsdict[EXT_PATH])

50
tests/test_settings/__init__.py
View File

@ -37,21 +37,22 @@ class SettingsAttributeTest(unittest.TestCase):
self.assertEqual(self.attribute.value, 'value')
self.assertEqual(self.attribute.priority, 10)
def test_set_per_key_priorities(self):
attribute = SettingsAttribute(
BaseSettings({'one': 10, 'two': 20}, 0), 0)
def test_overwrite_basesettings(self):
original_dict = {'one': 10, 'two': 20}
original_settings = BaseSettings(original_dict, 0)
attribute = SettingsAttribute(original_settings, 0)
new_dict = {'one': 11, 'two': 21}
new_dict = {'three': 11, 'four': 21}
attribute.set(new_dict, 10)
self.assertEqual(attribute.value['one'], 11)
self.assertEqual(attribute.value['two'], 21)
self.assertIsInstance(attribute.value, BaseSettings)
six.assertCountEqual(self, attribute.value, new_dict)
six.assertCountEqual(self, original_settings, original_dict)
new_settings = BaseSettings()
new_settings.set('one', 12, 20)
new_settings.set('two', 12, 0)
attribute.set(new_settings, 0)
self.assertEqual(attribute.value['one'], 12)
self.assertEqual(attribute.value['two'], 21)
new_settings = BaseSettings({'five': 12}, 0)
attribute.set(new_settings, 0) # Insufficient priority
six.assertCountEqual(self, attribute.value, new_dict)
attribute.set(new_settings, 10)
six.assertCountEqual(self, attribute.value, new_settings)
def test_repr(self):
self.assertEqual(repr(self.attribute),
@ -263,24 +264,15 @@ class BaseSettingsTest(unittest.TestCase):
self.assertEqual(settings.getpriority('key'), 99)
self.assertEqual(settings.getpriority('nonexistentkey'), None)
def test_getcomposite(self):
s = BaseSettings({'TEST_BASE': {1: 1, 2: 2},
def test_getwithbase(self):
s = BaseSettings({'TEST_BASE': BaseSettings({1: 1, 2: 2}, 'project'),
'TEST': BaseSettings({1: 10, 3: 30}, 'default'),
'HASNOBASE': BaseSettings({1: 1}, 'default')})
s['TEST'].set(4, 4, priority='project')
# When users specify a _BASE setting they explicitly don't want to use
# Scrapy's defaults, so we don't want to see anything that has a
# 'default' priority from TEST
cs = s._getcomposite('TEST')
self.assertEqual(len(cs), 3)
self.assertEqual(cs[1], 1)
self.assertEqual(cs[2], 2)
self.assertEqual(cs[4], 4)
cs = s._getcomposite('HASNOBASE')
self.assertEqual(len(cs), 1)
self.assertEqual(cs[1], 1)
cs = s._getcomposite('NONEXISTENT')
self.assertIsNone(cs)
'HASNOBASE': BaseSettings({3: 3000}, 'default')})
s['TEST'].set(2, 200, 'cmdline')
six.assertCountEqual(self, s.getwithbase('TEST'),
{1: 1, 2: 200, 3: 30})
six.assertCountEqual(self, s.getwithbase('HASNOBASE'), s['HASNOBASE'])
self.assertEqual(s.getwithbase('NONEXISTENT'), {})
def test_maxpriority(self):
# Empty settings should return 'default'

37
tests/test_utils_conf.py
View File

@ -8,46 +8,59 @@ class BuildComponentListTest(unittest.TestCase):
def test_build_dict(self):
d = {'one': 1, 'two': None, 'three': 8, 'four': 4}
self.assertEqual(build_component_list(d, lambda x: x),
self.assertEqual(build_component_list(d, convert=lambda x: x),
['one', 'four', 'three'])
def test_backwards_compatible_build_dict(self):
base = {'one': 1, 'two': 2, 'three': 3, 'five': 5, 'six': None}
custom = {'two': None, 'three': 8, 'four': 4}
self.assertEqual(build_component_list(base, custom,
convert=lambda x: x),
['one', 'four', 'five', 'three'])
def test_return_list(self):
custom = ['a', 'b', 'c']
self.assertEqual(build_component_list(custom, lambda x: x), custom)
self.assertEqual(build_component_list(None, custom,
convert=lambda x: x),
custom)
def test_map_dict(self):
custom = {'one': 1, 'two': 2, 'three': 3}
self.assertEqual(build_component_list(custom, lambda x: x.upper()),
self.assertEqual(build_component_list({}, custom,
convert=lambda x: x.upper()),
['ONE', 'TWO', 'THREE'])
def test_map_list(self):
custom = ['a', 'b', 'c']
self.assertEqual(build_component_list(custom, lambda x: x.upper()),
self.assertEqual(build_component_list(None, custom,
lambda x: x.upper()),
['A', 'B', 'C'])
def test_duplicate_components_in_dict(self):
duplicate_dict = {'one': 1, 'two': 2, 'ONE': 4}
self.assertRaises(ValueError,
build_component_list, duplicate_dict, lambda x: x.lower())
self.assertRaises(ValueError, build_component_list, {}, duplicate_dict,
convert=lambda x: x.lower())
def test_duplicate_components_in_list(self):
duplicate_list = ['a', 'b', 'a']
self.assertRaises(ValueError,
build_component_list, duplicate_list, lambda x: x)
self.assertRaises(ValueError, build_component_list, None,
duplicate_list, convert=lambda x: x)
def test_duplicate_components_in_basesettings(self):
# Higher priority takes precedence
duplicate_bs = BaseSettings({'one': 1, 'two': 2}, priority=0)
duplicate_bs.set('ONE', 4, priority=10)
self.assertEqual(build_component_list(duplicate_bs, convert=lambda x: x.lower()),
self.assertEqual(build_component_list(duplicate_bs,
convert=lambda x: x.lower()),
['two', 'one'])
duplicate_bs.set('one', duplicate_bs['one'], priority=20)
self.assertEqual(build_component_list(duplicate_bs, convert=lambda x: x.lower()),
self.assertEqual(build_component_list(duplicate_bs,
convert=lambda x: x.lower()),
['one', 'two'])
# Same priority raises ValueError
duplicate_bs.set('ONE', duplicate_bs['ONE'], priority=20)
self.assertRaises(ValueError,
build_component_list, duplicate_bs, convert=lambda x: x.lower())
self.assertRaises(ValueError, build_component_list, duplicate_bs,
convert=lambda x: x.lower())
class UtilsConfTestCase(unittest.TestCase):