diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index 08d8f3edf..cc0254d29 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -23,20 +23,22 @@ Here's an example:: 'myproject.middlewares.CustomDownloaderMiddleware': 543, } -The specified :setting:`DOWNLOADER_MIDDLEWARES` setting is merged with the -default one (i.e. it does not overwrite it) and then sorted by order to get the -final sorted list of enabled middlewares: the first middleware is the one -closer to the engine and the last is the one closer to the downloader. +The :setting:`DOWNLOADER_MIDDLEWARES` setting is merged with the +:setting:`DOWNLOADER_MIDDLEWARES_BASE` setting defined in Scrapy (and not meant +to be overridden) and then sorted by order to get the final sorted list of +enabled middlewares: the first middleware is the one closer to the engine and +the last is the one closer to the downloader. -To decide which order to assign to your middleware see the default -:setting:`DOWNLOADER_MIDDLEWARES` setting and pick a value according to +To decide which order to assign to your middleware see the +:setting:`DOWNLOADER_MIDDLEWARES_BASE` setting and pick a value according to where you want to insert the middleware. The order does matter because each middleware performs a different action and your middleware could depend on some previous (or subsequent) middleware being applied. -If you want to disable a built-in middleware you must define it in your -project's :setting:`DOWNLOADER_MIDDLEWARES` setting and assign ``None`` as its -value. For example, if you want to disable the user-agent middleware:: +If you want to disable a built-in middleware (the ones defined in +:setting:`DOWNLOADER_MIDDLEWARES_BASE` and enabled by default) you must define it +in your project's :setting:`DOWNLOADER_MIDDLEWARES` setting and assign `None` +as its value. For example, if you want to disable the user-agent middleware:: DOWNLOADER_MIDDLEWARES = { 'myproject.middlewares.CustomDownloaderMiddleware': 543, @@ -162,7 +164,7 @@ middleware, see the :ref:`downloader middleware usage guide `. For a list of the components enabled by default (and their orders) see the -:setting:`DOWNLOADER_MIDDLEWARES` setting. +:setting:`DOWNLOADER_MIDDLEWARES_BASE` setting. .. _cookies-mw: diff --git a/docs/topics/extensions.rst b/docs/topics/extensions.rst index 11c0aadb6..847353868 100644 --- a/docs/topics/extensions.rst +++ b/docs/topics/extensions.rst @@ -42,13 +42,14 @@ by a string: the full Python path to the extension's class name. For example:: As you can see, the :setting:`EXTENSIONS` setting is a dict where the keys are the extension paths, and their values are the orders, which define the -extension *loading* order. The specified :setting:`EXTENSIONS` setting is merged -with the default one (i.e. it does not overwrite it) and then sorted by order -to get the final sorted list of enabled extensions. +extension *loading* order. The :setting:`EXTENSIONS` setting is merged with the +:setting:`EXTENSIONS_BASE` setting defined in Scrapy (and not meant to be +overridden) and then sorted by order to get the final sorted list of enabled +extensions. As extensions typically do not depend on each other, their loading order is -irrelevant in most cases. This is why the default :setting:`EXTENSIONS` setting -defines all extensions with the same order (``500``). However, this feature can +irrelevant in most cases. This is why the :setting:`EXTENSIONS_BASE` setting +defines all extensions with the same order (``0``). However, this feature can be exploited if you need to add an extension which depends on other extensions already loaded. @@ -63,7 +64,7 @@ Disabling an extension ====================== In order to disable an extension that comes enabled by default (ie. those -included in the default :setting:`EXTENSIONS` setting) you must set its order to +included in the :setting:`EXTENSIONS_BASE` setting) you must set its order to ``None``. For example:: EXTENSIONS = { diff --git a/docs/topics/feed-exports.rst b/docs/topics/feed-exports.rst index d8b8da166..03c6fb3fb 100644 --- a/docs/topics/feed-exports.rst +++ b/docs/topics/feed-exports.rst @@ -265,6 +265,16 @@ Whether to export empty feeds (ie. feeds with no items). FEED_STORAGES ------------- +Default:: ``{}`` + +A dict containing additional feed storage backends supported by your project. +The keys are URI schemes and the values are paths to storage classes. + +.. setting:: FEED_STORAGES_BASE + +FEED_STORAGES_BASE +------------------ + Default:: { @@ -275,19 +285,30 @@ Default:: 'ftp': 'scrapy.extensions.feedexport.FTPFeedStorage', } -A dict containing all feed storage backends supported by your project. The keys -are URI schemes and the values are paths to storage classes. +A dict containing the built-in feed storage backends supported by Scrapy. You +can disable any of these backends by assigning ``None`` to their URI scheme in +:setting:`FEED_STORAGES`. E.g., to disable the built-in FTP storage backend +(without replacement), place this in your ``settings.py``:: -When you set :setting:`FEED_STORAGES` manually, e.g. in your project's settings -module, it will be merged with the default, not overwrite it. If you want to -disable any of the default feed storage backends, you must assign ``None`` as -their value. + FEED_STORAGES = { + 'ftp': None, + } .. setting:: FEED_EXPORTERS FEED_EXPORTERS -------------- +Default:: ``{}`` + +A dict containing additional exporters supported by your project. The keys are +serialization formats and the values are paths to :ref:`Item exporter +` classes. + +.. setting:: FEED_EXPORTERS_BASE + +FEED_EXPORTERS_BASE +------------------- Default:: { @@ -300,14 +321,14 @@ Default:: 'pickle': 'scrapy.exporters.PickleItemExporter', } -A dict containing all feed exporters supported by your project. The keys are -URI schemes and the values are paths to :ref:`Item exporter ` -classes. +A dict containing the built-in feed exporters supported by Scrapy. You can +disable any of these exporters by assigning ``None`` to their serialization +format in :setting:`FEED_EXPORTERS`. E.g., to disable the built-in CSV exporter +(without replacement), place this in your ``settings.py``:: -When you set :setting:`FEED_EXPORTERS` manually, e.g. in your project's settings -module, it will be merged with the default, not overwrite it. If you want to -disable any of the default feed exporters, you must assign ``None`` as their -value. + FEED_EXPORTERS = { + 'csv': None, + } .. _URI: http://en.wikipedia.org/wiki/Uniform_Resource_Identifier .. _Amazon S3: http://aws.amazon.com/s3/ diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 8908fae7e..aa0417e1a 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -269,11 +269,6 @@ Default:: The default headers used for Scrapy HTTP Requests. They're populated in the :class:`~scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware`. -When you set :setting:`DEFAULT_REQUEST_HEADERS` manually, e.g. in your -project's settings module, it will be merged with the default, not overwrite it. -If you want to disable any of the default request headers (and not replace them) -you must assign ``None`` as their value. - .. setting:: DEPTH_LIMIT DEPTH_LIMIT @@ -355,6 +350,16 @@ The downloader to use for crawling. DOWNLOADER_MIDDLEWARES ---------------------- +Default:: ``{}`` + +A dict containing the downloader middlewares enabled in your project, and their +orders. For more info see :ref:`topics-downloader-middleware-setting`. + +.. setting:: DOWNLOADER_MIDDLEWARES_BASE + +DOWNLOADER_MIDDLEWARES_BASE +--------------------------- + Default:: { @@ -375,16 +380,11 @@ Default:: 'scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware': 900, } -A dict containing the downloader middlewares enabled in your project, and their -orders. Low orders are closer to the engine, high orders are closer to the -downloader. - -When you set :setting:`DOWNLOADER_MIDDLEWARES` manually, e.g. in your project's -settings module, it will be merged with the default, not overwrite it. If you -want to disable any of the default downloader middlewares you must assign -``None`` as their value. - -For more info see :ref:`topics-downloader-middleware-setting`. +A dict containing the downloader middlewares enabled by default in Scrapy. Low +orders are closer to the engine, high orders are closer to the downloader. You +should never modify this setting in your project, modify +:setting:`DOWNLOADER_MIDDLEWARES` instead. For more info see +:ref:`topics-downloader-middleware-setting`. .. setting:: DOWNLOADER_STATS @@ -425,6 +425,16 @@ spider attribute. DOWNLOAD_HANDLERS ----------------- +Default: ``{}`` + +A dict containing the request downloader handlers enabled in your project. +See :setting:`DOWNLOAD_HANDLERS_BASE` for example format. + +.. setting:: DOWNLOAD_HANDLERS_BASE + +DOWNLOAD_HANDLERS_BASE +---------------------- + Default:: { @@ -436,15 +446,16 @@ Default:: } -A dict containing the request downloader handlers enabled in your project. +A dict containing the request download handlers enabled by default in Scrapy. +You should never modify this setting in your project, modify +:setting:`DOWNLOAD_HANDLERS` instead. -When you set :setting:`DOWNLOAD_HANDLERS` manually, e.g. in your project's -settings module, it will be merged with the default, not overwrite it. If you -want to disable any of the default download handlers you must assign ``None`` -as their value. For example, if you want to disable the file download handler:: +You can disable any of these download handlers by assigning ``None`` to their +URI scheme in :setting:`DOWNLOAD_HANDLERS`. E.g., to disable the built-in FTP +handler (without replacement), place this in your ``settings.py``:: DOWNLOAD_HANDLERS = { - 'file': None, + 'ftp': None, } .. setting:: DOWNLOAD_TIMEOUT @@ -544,6 +555,15 @@ to ``vi`` (on Unix systems) or the IDLE editor (on Windows). EXTENSIONS ---------- +Default:: ``{}`` + +A dict containing the extensions enabled in your project, and their orders. + +.. setting:: EXTENSIONS_BASE + +EXTENSIONS_BASE +--------------- + Default:: { @@ -558,15 +578,10 @@ Default:: 'scrapy.extensions.throttle.AutoThrottle': 0, } -A dict containing the extensions enabled in your project, and their orders. By -default, this setting contains all stable built-in extensions. Keep in mind that +A dict containing the extensions available by default in Scrapy, and their +orders. This setting contains all stable built-in extensions. Keep in mind that some of them need to be enabled through a setting. -When you set :setting:`EXTENSIONS` manually, e.g. in your project's settings -module, it will be merged with the default, not overwrite it. If you want to -disable any of the default enabled extensions you must assign ``None`` as their -value. - For more information See the :ref:`extensions user guide ` and the :ref:`list of available extensions `. @@ -589,6 +604,16 @@ Example:: 'mybot.pipelines.validate.StoreMyItem': 800, } +.. setting:: ITEM_PIPELINES_BASE + +ITEM_PIPELINES_BASE +------------------- + +Default: ``{}`` + +A dict containing the pipelines enabled by default in Scrapy. You should never +modify this setting in your project, modify :setting:`ITEM_PIPELINES` instead. + .. setting:: LOG_ENABLED LOG_ENABLED @@ -878,6 +903,16 @@ The scheduler to use for crawling. SPIDER_CONTRACTS ---------------- +Default:: ``{}`` + +A dict containing the spider contracts enabled in your project, used for +testing spiders. For more info see :ref:`topics-contracts`. + +.. setting:: SPIDER_CONTRACTS_BASE + +SPIDER_CONTRACTS_BASE +--------------------- + Default:: { @@ -886,13 +921,17 @@ Default:: 'scrapy.contracts.default.ScrapesContract': 3, } -A dict containing the scrapy contracts enabled in your project, used for -testing spiders. For more info see :ref:`topics-contracts`. +A dict containing the scrapy contracts enabled by default in Scrapy. You should +never modify this setting in your project, modify :setting:`SPIDER_CONTRACTS` +instead. For more info see :ref:`topics-contracts`. -When you set :setting:`SPIDER_CONTRACTS` manually, e.g. in your project's -settings module, it will be merged with the default, not overwrite it. If you -want to disable any of the default contracts you must assign ``None`` as their -value. +You can disable any of these contracts by assigning ``None`` to their class +path in :setting:`SPIDER_CONTRACTS`. E.g., to disable the built-in +``ScrapesContract``, place this in your ``settings.py``:: + + SPIDER_CONTRACTS = { + 'scrapy.contracts.default.ScrapesContract': None, + } .. setting:: SPIDER_LOADER_CLASS @@ -909,6 +948,16 @@ The class that will be used for loading spiders, which must implement the SPIDER_MIDDLEWARES ------------------ +Default:: ``{}`` + +A dict containing the spider middlewares enabled in your project, and their +orders. For more info see :ref:`topics-spider-middleware-setting`. + +.. setting:: SPIDER_MIDDLEWARES_BASE + +SPIDER_MIDDLEWARES_BASE +----------------------- + Default:: { @@ -919,14 +968,9 @@ Default:: 'scrapy.spidermiddlewares.depth.DepthMiddleware': 900, } -A dict containing the spider middlewares enabled in your project, and their -orders. Low orders are closer to the engine, high orders are closer to the -spider. For more info see :ref:`topics-spider-middleware-setting`. - -When you set :setting:`SPIDER_MIDDLEWARES` manually, e.g. in your project's -settings module, it will be merged with the default, not overwrite it. If you -want to disable any of the default spider middlewares you must assign ``None`` -as their value. +A dict containing the spider middlewares enabled by default in Scrapy, and +their orders. Low orders are closer to the engine, high orders are closer to +the spider. For more info see :ref:`topics-spider-middleware-setting`. .. setting:: SPIDER_MODULES diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst index d448801d3..84daaaa55 100644 --- a/docs/topics/spider-middleware.rst +++ b/docs/topics/spider-middleware.rst @@ -24,20 +24,22 @@ Here's an example:: 'myproject.middlewares.CustomSpiderMiddleware': 543, } -The specified :setting:`SPIDER_MIDDLEWARES` setting is merged with the default -one (i.e. it does not overwrite it) and then sorted by order to get the final -sorted list of enabled middlewares: the first middleware is the one closer to -the engine and the last is the one closer to the spider. +The :setting:`SPIDER_MIDDLEWARES` setting is merged with the +:setting:`SPIDER_MIDDLEWARES_BASE` setting defined in Scrapy (and not meant to +be overridden) and then sorted by order to get the final sorted list of enabled +middlewares: the first middleware is the one closer to the engine and the last +is the one closer to the spider. -To decide which order to assign to your middleware see the default -:setting:`SPIDER_MIDDLEWARES` setting and pick a value according to where +To decide which order to assign to your middleware see the +:setting:`SPIDER_MIDDLEWARES_BASE` setting and pick a value according to where you want to insert the middleware. The order does matter because each middleware performs a different action and your middleware could depend on some previous (or subsequent) middleware being applied. -If you want to disable a builtin middleware you must define it in your project's -:setting:`SPIDER_MIDDLEWARES` setting and assign ``None`` as its value. For -example, if you want to disable the off-site middleware:: +If you want to disable a builtin middleware (the ones defined in +:setting:`SPIDER_MIDDLEWARES_BASE`, and enabled by default) you must define it +in your project :setting:`SPIDER_MIDDLEWARES` setting and assign `None` as its +value. For example, if you want to disable the off-site middleware:: SPIDER_MIDDLEWARES = { 'myproject.middlewares.CustomSpiderMiddleware': 543, @@ -171,7 +173,7 @@ information on how to use them and how to write your own spider middleware, see the :ref:`spider middleware usage guide `. For a list of the components enabled by default (and their orders) see the -:setting:`SPIDER_MIDDLEWARES` setting. +:setting:`SPIDER_MIDDLEWARES_BASE` setting. DepthMiddleware --------------- diff --git a/scrapy/commands/check.py b/scrapy/commands/check.py index a423ba2c9..b8a9ef989 100644 --- a/scrapy/commands/check.py +++ b/scrapy/commands/check.py @@ -58,7 +58,7 @@ class Command(ScrapyCommand): def run(self, args, opts): # load contracts - contracts = build_component_list(self.settings._getcomposite('SPIDER_CONTRACTS')) + contracts = build_component_list(self.settings.getwithbase('SPIDER_CONTRACTS')) conman = ContractsManager(load_object(c) for c in contracts) runner = TextTestRunner(verbosity=2 if opts.verbose else 1) result = TextTestResult(runner.stream, runner.descriptions, runner.verbosity) diff --git a/scrapy/commands/crawl.py b/scrapy/commands/crawl.py index 7f5c64c20..4b986bf9d 100644 --- a/scrapy/commands/crawl.py +++ b/scrapy/commands/crawl.py @@ -35,7 +35,8 @@ class Command(ScrapyCommand): self.settings.set('FEED_URI', 'stdout:', priority='cmdline') else: self.settings.set('FEED_URI', opts.output, priority='cmdline') - feed_exporters = without_none_values(self.settings._getcomposite('FEED_EXPORTERS')) + feed_exporters = without_none_values( + self.settings.getwithbase('FEED_EXPORTERS')) valid_output_formats = feed_exporters.keys() if not opts.output_format: opts.output_format = os.path.splitext(opts.output)[1].replace(".", "") diff --git a/scrapy/commands/runspider.py b/scrapy/commands/runspider.py index 72229bcf5..1da09e4da 100644 --- a/scrapy/commands/runspider.py +++ b/scrapy/commands/runspider.py @@ -58,7 +58,7 @@ class Command(ScrapyCommand): self.settings.set('FEED_URI', 'stdout:', priority='cmdline') else: self.settings.set('FEED_URI', opts.output, priority='cmdline') - feed_exporters = without_none_values(self.settings._getcomposite('FEED_EXPORTERS')) + feed_exporters = without_none_values(self.settings.getwithbase('FEED_EXPORTERS')) valid_output_formats = feed_exporters.keys() if not opts.output_format: opts.output_format = os.path.splitext(opts.output)[1].replace(".", "") diff --git a/scrapy/core/downloader/handlers/__init__.py b/scrapy/core/downloader/handlers/__init__.py index 0e78e04f4..bc5cd742e 100644 --- a/scrapy/core/downloader/handlers/__init__.py +++ b/scrapy/core/downloader/handlers/__init__.py @@ -20,7 +20,8 @@ class DownloadHandlers(object): self._schemes = {} # stores acceptable schemes on instancing self._handlers = {} # stores instanced handlers for schemes self._notconfigured = {} # remembers failed handlers - handlers = without_none_values(crawler.settings._getcomposite('DOWNLOAD_HANDLERS')) + handlers = without_none_values( + crawler.settings.getwithbase('DOWNLOAD_HANDLERS')) for scheme, clspath in six.iteritems(handlers): self._schemes[scheme] = clspath diff --git a/scrapy/core/downloader/middleware.py b/scrapy/core/downloader/middleware.py index 958113fc3..c3b23e284 100644 --- a/scrapy/core/downloader/middleware.py +++ b/scrapy/core/downloader/middleware.py @@ -19,7 +19,8 @@ class DownloaderMiddlewareManager(MiddlewareManager): @classmethod def _get_mwlist_from_settings(cls, settings): - return build_component_list(settings._getcomposite('DOWNLOADER_MIDDLEWARES')) + return build_component_list( + settings.getwithbase('DOWNLOADER_MIDDLEWARES')) def _add_middleware(self, mw): if hasattr(mw, 'process_request'): diff --git a/scrapy/core/spidermw.py b/scrapy/core/spidermw.py index b5c80c350..a206e4b0c 100644 --- a/scrapy/core/spidermw.py +++ b/scrapy/core/spidermw.py @@ -18,7 +18,7 @@ class SpiderMiddlewareManager(MiddlewareManager): @classmethod def _get_mwlist_from_settings(cls, settings): - return build_component_list(settings._getcomposite('SPIDER_MIDDLEWARES')) + return build_component_list(settings.getwithbase('SPIDER_MIDDLEWARES')) def _add_middleware(self, mw): super(SpiderMiddlewareManager, self)._add_middleware(mw) diff --git a/scrapy/extension.py b/scrapy/extension.py index 4ceb32c68..e39e456fa 100644 --- a/scrapy/extension.py +++ b/scrapy/extension.py @@ -12,4 +12,4 @@ class ExtensionManager(MiddlewareManager): @classmethod def _get_mwlist_from_settings(cls, settings): - return build_component_list(settings._getcomposite('EXTENSIONS')) + return build_component_list(settings.getwithbase('EXTENSIONS')) diff --git a/scrapy/extensions/feedexport.py b/scrapy/extensions/feedexport.py index 1e27a1e7e..daea551cb 100644 --- a/scrapy/extensions/feedexport.py +++ b/scrapy/extensions/feedexport.py @@ -196,7 +196,7 @@ class FeedExporter(object): return item def _load_components(self, setting_prefix): - conf = without_none_values(self.settings._getcomposite(setting_prefix)) + conf = without_none_values(self.settings.getwithbase(setting_prefix)) d = {} for k, v in conf.items(): try: diff --git a/scrapy/pipelines/__init__.py b/scrapy/pipelines/__init__.py index 8df0d3154..2ef8786d0 100644 --- a/scrapy/pipelines/__init__.py +++ b/scrapy/pipelines/__init__.py @@ -13,7 +13,7 @@ class ItemPipelineManager(MiddlewareManager): @classmethod def _get_mwlist_from_settings(cls, settings): - return build_component_list(settings._getcomposite('ITEM_PIPELINES')) + return build_component_list(settings.getwithbase('ITEM_PIPELINES')) def _add_middleware(self, pipe): super(ItemPipelineManager, self)._add_middleware(pipe) diff --git a/scrapy/settings/__init__.py b/scrapy/settings/__init__.py index 13656298b..342d2585e 100644 --- a/scrapy/settings/__init__.py +++ b/scrapy/settings/__init__.py @@ -49,14 +49,11 @@ class SettingsAttribute(object): def set(self, value, priority): """Sets value if priority is higher or equal than current priority.""" - if isinstance(self.value, BaseSettings): - # Ignore self.priority if self.value has per-key priorities - self.value.update(value, priority) - self.priority = max(self.value.maxpriority(), priority) - else: - if priority >= self.priority: - self.value = value - self.priority = priority + if priority >= self.priority: + if isinstance(self.value, BaseSettings): + value = BaseSettings(value, priority=priority) + self.value = value + self.priority = priority def __str__(self): return " get_settings_priority('default'): - compsett.set(k, self[name][k], prio) - return compsett - return self[name] + def getwithbase(self, name): + """Get a composition of a dictionary-like setting and its `_BASE` + counterpart. + + :param name: name of the dictionary-like setting + :type name: string + """ + compbs = BaseSettings() + compbs.update(self[name + '_BASE']) + compbs.update(self[name]) + return compbs def getpriority(self, name): """ @@ -223,10 +211,9 @@ class BaseSettings(MutableMapping): :param name: the setting name :type name: string """ - prio = None - if name in self: - prio = self.attributes[name].priority - return prio + if name not in self: + return None + return self.attributes[name].priority def maxpriority(self): """ diff --git a/scrapy/settings/default_settings.py b/scrapy/settings/default_settings.py index 375efcdbb..8435b0354 100644 --- a/scrapy/settings/default_settings.py +++ b/scrapy/settings/default_settings.py @@ -63,7 +63,8 @@ DNS_TIMEOUT = 60 DOWNLOAD_DELAY = 0 -DOWNLOAD_HANDLERS = { +DOWNLOAD_HANDLERS = {} +DOWNLOAD_HANDLERS_BASE = { 'file': 'scrapy.core.downloader.handlers.file.FileDownloadHandler', 'http': 'scrapy.core.downloader.handlers.http.HTTPDownloadHandler', 'https': 'scrapy.core.downloader.handlers.http.HTTPDownloadHandler', @@ -81,7 +82,9 @@ DOWNLOADER = 'scrapy.core.downloader.Downloader' DOWNLOADER_HTTPCLIENTFACTORY = 'scrapy.core.downloader.webclient.ScrapyHTTPClientFactory' DOWNLOADER_CLIENTCONTEXTFACTORY = 'scrapy.core.downloader.contextfactory.ScrapyClientContextFactory' -DOWNLOADER_MIDDLEWARES = { +DOWNLOADER_MIDDLEWARES = {} + +DOWNLOADER_MIDDLEWARES_BASE = { # Engine side 'scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware': 100, 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware': 300, @@ -113,7 +116,9 @@ except KeyError: else: EDITOR = 'vi' -EXTENSIONS = { +EXTENSIONS = {} + +EXTENSIONS_BASE = { 'scrapy.extensions.corestats.CoreStats': 0, 'scrapy.extensions.telnet.TelnetConsole': 0, 'scrapy.extensions.memusage.MemoryUsage': 0, @@ -130,14 +135,16 @@ FEED_URI_PARAMS = None # a function to extend uri arguments FEED_FORMAT = 'jsonlines' FEED_STORE_EMPTY = False FEED_EXPORT_FIELDS = None -FEED_STORAGES = { +FEED_STORAGES = {} +FEED_STORAGES_BASE = { '': 'scrapy.extensions.feedexport.FileFeedStorage', 'file': 'scrapy.extensions.feedexport.FileFeedStorage', 'stdout': 'scrapy.extensions.feedexport.StdoutFeedStorage', 's3': 'scrapy.extensions.feedexport.S3FeedStorage', 'ftp': 'scrapy.extensions.feedexport.FTPFeedStorage', } -FEED_EXPORTERS = { +FEED_EXPORTERS = {} +FEED_EXPORTERS_BASE = { 'json': 'scrapy.exporters.JsonItemExporter', 'jsonlines': 'scrapy.exporters.JsonLinesItemExporter', 'jl': 'scrapy.exporters.JsonLinesItemExporter', @@ -163,6 +170,7 @@ HTTPCACHE_GZIP = False ITEM_PROCESSOR = 'scrapy.pipelines.ItemPipelineManager' ITEM_PIPELINES = {} +ITEM_PIPELINES_BASE = {} LOG_ENABLED = True LOG_ENCODING = 'utf-8' @@ -221,7 +229,9 @@ SCHEDULER_MEMORY_QUEUE = 'scrapy.squeues.LifoMemoryQueue' SPIDER_LOADER_CLASS = 'scrapy.spiderloader.SpiderLoader' -SPIDER_MIDDLEWARES = { +SPIDER_MIDDLEWARES = {} + +SPIDER_MIDDLEWARES_BASE = { # Engine side 'scrapy.spidermiddlewares.httperror.HttpErrorMiddleware': 50, 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware': 500, @@ -248,7 +258,8 @@ TELNETCONSOLE_ENABLED = 1 TELNETCONSOLE_PORT = [6023, 6073] TELNETCONSOLE_HOST = '127.0.0.1' -SPIDER_CONTRACTS = { +SPIDER_CONTRACTS = {} +SPIDER_CONTRACTS_BASE = { 'scrapy.contracts.default.UrlContract': 1, 'scrapy.contracts.default.ReturnsContract': 2, 'scrapy.contracts.default.ScrapesContract': 3, diff --git a/scrapy/utils/conf.py b/scrapy/utils/conf.py index 57f2b6322..e8af90f11 100644 --- a/scrapy/utils/conf.py +++ b/scrapy/utils/conf.py @@ -10,7 +10,7 @@ from scrapy.utils.deprecate import update_classpath from scrapy.utils.python import without_none_values -def build_component_list(compdict, convert=update_classpath): +def build_component_list(compdict, custom=None, convert=update_classpath): """Compose a component list from a { class: order } dictionary.""" def _check_components(complist): @@ -34,9 +34,15 @@ def build_component_list(compdict, convert=update_classpath): _check_components(compdict) return {convert(k): v for k, v in six.iteritems(compdict)} - if isinstance(compdict, (list, tuple)): - _check_components(compdict) - return type(compdict)(convert(c) for c in compdict) + # BEGIN Backwards compatibility for old (base, custom) call signature + if isinstance(custom, (list, tuple)): + _check_components(custom) + return type(custom)(convert(c) for c in custom) + + if custom is not None: + compdict.update(custom) + # END Backwards compatibility + compdict = without_none_values(_map_keys(compdict)) return [k for k, v in sorted(six.iteritems(compdict), key=itemgetter(1))] diff --git a/tests/test_cmdline/__init__.py b/tests/test_cmdline/__init__.py index 5192fb0fa..c2de4fbc8 100644 --- a/tests/test_cmdline/__init__.py +++ b/tests/test_cmdline/__init__.py @@ -1,10 +1,11 @@ -import os import json -import sys -import shutil +import os import pstats -import tempfile +import shutil +import six from subprocess import Popen, PIPE +import sys +import tempfile import unittest try: from cStringIO import StringIO @@ -57,14 +58,14 @@ class CmdlineTest(unittest.TestCase): shutil.rmtree(path) def test_override_dict_settings(self): + EXT_PATH = "tests.test_cmdline.extensions.DummyExtension" + EXTENSIONS = {EXT_PATH: 200} settingsstr = self._execute('settings', '--get', 'EXTENSIONS', '-s', - ('EXTENSIONS={"tests.test_cmdline.extensions.TestExtension": ' - '100, "tests.test_cmdline.extensions.DummyExtension": 200}')) + 'EXTENSIONS=' + json.dumps(EXTENSIONS)) # XXX: There's gotta be a smarter way to do this... self.assertNotIn("...", settingsstr) for char in ("'", "<", ">", 'u"'): settingsstr = settingsstr.replace(char, '"') settingsdict = json.loads(settingsstr) - self.assertIn('tests.test_cmdline.extensions.DummyExtension', settingsdict) - self.assertIn('value=200', settingsdict['tests.test_cmdline.extensions.DummyExtension']) - self.assertIn('value=100', settingsdict['tests.test_cmdline.extensions.TestExtension']) + six.assertCountEqual(self, settingsdict.keys(), EXTENSIONS.keys()) + self.assertIn('value=200', settingsdict[EXT_PATH]) diff --git a/tests/test_settings/__init__.py b/tests/test_settings/__init__.py index 4ef08bb0b..44b9b6df3 100644 --- a/tests/test_settings/__init__.py +++ b/tests/test_settings/__init__.py @@ -37,21 +37,22 @@ class SettingsAttributeTest(unittest.TestCase): self.assertEqual(self.attribute.value, 'value') self.assertEqual(self.attribute.priority, 10) - def test_set_per_key_priorities(self): - attribute = SettingsAttribute( - BaseSettings({'one': 10, 'two': 20}, 0), 0) + def test_overwrite_basesettings(self): + original_dict = {'one': 10, 'two': 20} + original_settings = BaseSettings(original_dict, 0) + attribute = SettingsAttribute(original_settings, 0) - new_dict = {'one': 11, 'two': 21} + new_dict = {'three': 11, 'four': 21} attribute.set(new_dict, 10) - self.assertEqual(attribute.value['one'], 11) - self.assertEqual(attribute.value['two'], 21) + self.assertIsInstance(attribute.value, BaseSettings) + six.assertCountEqual(self, attribute.value, new_dict) + six.assertCountEqual(self, original_settings, original_dict) - new_settings = BaseSettings() - new_settings.set('one', 12, 20) - new_settings.set('two', 12, 0) - attribute.set(new_settings, 0) - self.assertEqual(attribute.value['one'], 12) - self.assertEqual(attribute.value['two'], 21) + new_settings = BaseSettings({'five': 12}, 0) + attribute.set(new_settings, 0) # Insufficient priority + six.assertCountEqual(self, attribute.value, new_dict) + attribute.set(new_settings, 10) + six.assertCountEqual(self, attribute.value, new_settings) def test_repr(self): self.assertEqual(repr(self.attribute), @@ -263,24 +264,15 @@ class BaseSettingsTest(unittest.TestCase): self.assertEqual(settings.getpriority('key'), 99) self.assertEqual(settings.getpriority('nonexistentkey'), None) - def test_getcomposite(self): - s = BaseSettings({'TEST_BASE': {1: 1, 2: 2}, + def test_getwithbase(self): + s = BaseSettings({'TEST_BASE': BaseSettings({1: 1, 2: 2}, 'project'), 'TEST': BaseSettings({1: 10, 3: 30}, 'default'), - 'HASNOBASE': BaseSettings({1: 1}, 'default')}) - s['TEST'].set(4, 4, priority='project') - # When users specify a _BASE setting they explicitly don't want to use - # Scrapy's defaults, so we don't want to see anything that has a - # 'default' priority from TEST - cs = s._getcomposite('TEST') - self.assertEqual(len(cs), 3) - self.assertEqual(cs[1], 1) - self.assertEqual(cs[2], 2) - self.assertEqual(cs[4], 4) - cs = s._getcomposite('HASNOBASE') - self.assertEqual(len(cs), 1) - self.assertEqual(cs[1], 1) - cs = s._getcomposite('NONEXISTENT') - self.assertIsNone(cs) + 'HASNOBASE': BaseSettings({3: 3000}, 'default')}) + s['TEST'].set(2, 200, 'cmdline') + six.assertCountEqual(self, s.getwithbase('TEST'), + {1: 1, 2: 200, 3: 30}) + six.assertCountEqual(self, s.getwithbase('HASNOBASE'), s['HASNOBASE']) + self.assertEqual(s.getwithbase('NONEXISTENT'), {}) def test_maxpriority(self): # Empty settings should return 'default' diff --git a/tests/test_utils_conf.py b/tests/test_utils_conf.py index af15d3184..dab41ac8d 100644 --- a/tests/test_utils_conf.py +++ b/tests/test_utils_conf.py @@ -8,46 +8,59 @@ class BuildComponentListTest(unittest.TestCase): def test_build_dict(self): d = {'one': 1, 'two': None, 'three': 8, 'four': 4} - self.assertEqual(build_component_list(d, lambda x: x), + self.assertEqual(build_component_list(d, convert=lambda x: x), ['one', 'four', 'three']) + def test_backwards_compatible_build_dict(self): + base = {'one': 1, 'two': 2, 'three': 3, 'five': 5, 'six': None} + custom = {'two': None, 'three': 8, 'four': 4} + self.assertEqual(build_component_list(base, custom, + convert=lambda x: x), + ['one', 'four', 'five', 'three']) + def test_return_list(self): custom = ['a', 'b', 'c'] - self.assertEqual(build_component_list(custom, lambda x: x), custom) + self.assertEqual(build_component_list(None, custom, + convert=lambda x: x), + custom) def test_map_dict(self): custom = {'one': 1, 'two': 2, 'three': 3} - self.assertEqual(build_component_list(custom, lambda x: x.upper()), + self.assertEqual(build_component_list({}, custom, + convert=lambda x: x.upper()), ['ONE', 'TWO', 'THREE']) def test_map_list(self): custom = ['a', 'b', 'c'] - self.assertEqual(build_component_list(custom, lambda x: x.upper()), + self.assertEqual(build_component_list(None, custom, + lambda x: x.upper()), ['A', 'B', 'C']) def test_duplicate_components_in_dict(self): duplicate_dict = {'one': 1, 'two': 2, 'ONE': 4} - self.assertRaises(ValueError, - build_component_list, duplicate_dict, lambda x: x.lower()) + self.assertRaises(ValueError, build_component_list, {}, duplicate_dict, + convert=lambda x: x.lower()) def test_duplicate_components_in_list(self): duplicate_list = ['a', 'b', 'a'] - self.assertRaises(ValueError, - build_component_list, duplicate_list, lambda x: x) + self.assertRaises(ValueError, build_component_list, None, + duplicate_list, convert=lambda x: x) def test_duplicate_components_in_basesettings(self): # Higher priority takes precedence duplicate_bs = BaseSettings({'one': 1, 'two': 2}, priority=0) duplicate_bs.set('ONE', 4, priority=10) - self.assertEqual(build_component_list(duplicate_bs, convert=lambda x: x.lower()), + self.assertEqual(build_component_list(duplicate_bs, + convert=lambda x: x.lower()), ['two', 'one']) duplicate_bs.set('one', duplicate_bs['one'], priority=20) - self.assertEqual(build_component_list(duplicate_bs, convert=lambda x: x.lower()), + self.assertEqual(build_component_list(duplicate_bs, + convert=lambda x: x.lower()), ['one', 'two']) # Same priority raises ValueError duplicate_bs.set('ONE', duplicate_bs['ONE'], priority=20) - self.assertRaises(ValueError, - build_component_list, duplicate_bs, convert=lambda x: x.lower()) + self.assertRaises(ValueError, build_component_list, duplicate_bs, + convert=lambda x: x.lower()) class UtilsConfTestCase(unittest.TestCase):