diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index 9122e5cb5..08d8f3edf 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -23,22 +23,20 @@ Here's an example:: 'myproject.middlewares.CustomDownloaderMiddleware': 543, } -The :setting:`DOWNLOADER_MIDDLEWARES` setting is merged with the -:setting:`DOWNLOADER_MIDDLEWARES_BASE` setting defined in Scrapy (and not meant to -be overridden) and then sorted by order to get the final sorted list of enabled -middlewares: the first middleware is the one closer to the engine and the last -is the one closer to the downloader. +The specified :setting:`DOWNLOADER_MIDDLEWARES` setting is merged with the +default one (i.e. it does not overwrite it) and then sorted by order to get the +final sorted list of enabled middlewares: the first middleware is the one +closer to the engine and the last is the one closer to the downloader. -To decide which order to assign to your middleware see the -:setting:`DOWNLOADER_MIDDLEWARES_BASE` setting and pick a value according to +To decide which order to assign to your middleware see the default +:setting:`DOWNLOADER_MIDDLEWARES` setting and pick a value according to where you want to insert the middleware. The order does matter because each middleware performs a different action and your middleware could depend on some previous (or subsequent) middleware being applied. -If you want to disable a built-in middleware (the ones defined in -:setting:`DOWNLOADER_MIDDLEWARES_BASE` and enabled by default) you must define it -in your project's :setting:`DOWNLOADER_MIDDLEWARES` setting and assign `None` -as its value. For example, if you want to disable the user-agent middleware:: +If you want to disable a built-in middleware you must define it in your +project's :setting:`DOWNLOADER_MIDDLEWARES` setting and assign ``None`` as its +value. For example, if you want to disable the user-agent middleware:: DOWNLOADER_MIDDLEWARES = { 'myproject.middlewares.CustomDownloaderMiddleware': 543, @@ -164,7 +162,7 @@ middleware, see the :ref:`downloader middleware usage guide `. For a list of the components enabled by default (and their orders) see the -:setting:`DOWNLOADER_MIDDLEWARES_BASE` setting. +:setting:`DOWNLOADER_MIDDLEWARES` setting. .. _cookies-mw: diff --git a/docs/topics/extensions.rst b/docs/topics/extensions.rst index fb5220e9d..a71b8bcee 100644 --- a/docs/topics/extensions.rst +++ b/docs/topics/extensions.rst @@ -42,17 +42,15 @@ by a string: the full Python path to the extension's class name. For example:: As you can see, the :setting:`EXTENSIONS` setting is a dict where the keys are the extension paths, and their values are the orders, which define the -extension *loading* order. Extensions orders are not as important as middleware -orders though, and they are typically irrelevant, ie. it doesn't matter in -which order the extensions are loaded because they don't depend on each other -[1]. +extension *loading* order. The specified :setting:`EXTENSIONS` setting is merged +with the default one (i.e. it does not overwrite it) and then sorted by order +to get the final sorted list of enabled extensions. -However, this feature can be exploited if you need to add an extension which -depends on other extensions already loaded. - -[1] This is is why the :setting:`EXTENSIONS_BASE` setting in Scrapy (which -contains all built-in extensions enabled by default) defines all the extensions -with the same order (``500``). +As extensions typically do not depend on each other, their loading order is +irrelevant in most cases. This is why the default :setting:`EXTENSIONS` setting +defines all extensions with the same order (``500``). However, this feature can +be exploited if you need to add an extension which depends on other extensions +already loaded. Available, enabled and disabled extensions ========================================== @@ -65,7 +63,7 @@ Disabling an extension ====================== In order to disable an extension that comes enabled by default (ie. those -included in the :setting:`EXTENSIONS_BASE` setting) you must set its order to +included in the default :setting:`EXTENSIONS` setting) you must set its order to ``None``. For example:: EXTENSIONS = { diff --git a/docs/topics/feed-exports.rst b/docs/topics/feed-exports.rst index d9444e34a..d8b8da166 100644 --- a/docs/topics/feed-exports.rst +++ b/docs/topics/feed-exports.rst @@ -265,16 +265,6 @@ Whether to export empty feeds (ie. feeds with no items). FEED_STORAGES ------------- -Default:: ``{}`` - -A dict containing additional feed storage backends supported by your project. -The keys are URI schemes and the values are paths to storage classes. - -.. setting:: FEED_STORAGES_BASE - -FEED_STORAGES_BASE ------------------- - Default:: { @@ -285,36 +275,39 @@ Default:: 'ftp': 'scrapy.extensions.feedexport.FTPFeedStorage', } -A dict containing the built-in feed storage backends supported by Scrapy. +A dict containing all feed storage backends supported by your project. The keys +are URI schemes and the values are paths to storage classes. + +When you set :setting:`FEED_STORAGES` manually, e.g. in your project's settings +module, it will be merged with the default, not overwrite it. If you want to +disable any of the default feed storage backends, you must assign ``None`` as +their value. .. setting:: FEED_EXPORTERS FEED_EXPORTERS -------------- -Default:: ``{}`` - -A dict containing additional exporters supported by your project. The keys are -URI schemes and the values are paths to :ref:`Item exporter ` -classes. - -.. setting:: FEED_EXPORTERS_BASE - -FEED_EXPORTERS_BASE -------------------- - Default:: - FEED_EXPORTERS_BASE = { + { 'json': 'scrapy.exporters.JsonItemExporter', 'jsonlines': 'scrapy.exporters.JsonLinesItemExporter', + 'jl': 'scrapy.exporters.JsonLinesItemExporter', 'csv': 'scrapy.exporters.CsvItemExporter', 'xml': 'scrapy.exporters.XmlItemExporter', 'marshal': 'scrapy.exporters.MarshalItemExporter', + 'pickle': 'scrapy.exporters.PickleItemExporter', } -A dict containing the built-in feed exporters supported by Scrapy. +A dict containing all feed exporters supported by your project. The keys are +URI schemes and the values are paths to :ref:`Item exporter ` +classes. +When you set :setting:`FEED_EXPORTERS` manually, e.g. in your project's settings +module, it will be merged with the default, not overwrite it. If you want to +disable any of the default feed exporters, you must assign ``None`` as their +value. .. _URI: http://en.wikipedia.org/wiki/Uniform_Resource_Identifier .. _Amazon S3: http://aws.amazon.com/s3/ diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 484065406..642f4eb84 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -269,6 +269,11 @@ Default:: The default headers used for Scrapy HTTP Requests. They're populated in the :class:`~scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware`. +When you set :setting:`DEFAULT_REQUEST_HEADERS` manually, e.g. in your +project's settings module, it will be merged with the default, not overwrite it. +If you want to disable any of the default request headers (and not replace them) +you must assign ``None`` as their value. + .. setting:: DEPTH_LIMIT DEPTH_LIMIT @@ -350,16 +355,6 @@ The downloader to use for crawling. DOWNLOADER_MIDDLEWARES ---------------------- -Default:: ``{}`` - -A dict containing the downloader middlewares enabled in your project, and their -orders. For more info see :ref:`topics-downloader-middleware-setting`. - -.. setting:: DOWNLOADER_MIDDLEWARES_BASE - -DOWNLOADER_MIDDLEWARES_BASE ---------------------------- - Default:: { @@ -369,6 +364,7 @@ Default:: 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': 400, 'scrapy.downloadermiddlewares.retry.RetryMiddleware': 500, 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware': 550, + 'scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware': 560, 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware': 580, 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 590, 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware': 600, @@ -379,10 +375,16 @@ Default:: 'scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware': 900, } -A dict containing the downloader middlewares enabled by default in Scrapy. You -should never modify this setting in your project, modify -:setting:`DOWNLOADER_MIDDLEWARES` instead. For more info see -:ref:`topics-downloader-middleware-setting`. +A dict containing the downloader middlewares enabled in your project, and their +orders. Low orders are closer to the engine, high orders are closer to the +downloader. + +When you set :setting:`DOWNLOADER_MIDDLEWARES` manually, e.g. in your project's +settings module, it will be merged with the default, not overwrite it. If you +want to disable any of the default downloader middlewares you must assign +``None`` as their value. + +For more info see :ref:`topics-downloader-middleware-setting`. .. setting:: DOWNLOADER_STATS @@ -423,33 +425,23 @@ spider attribute. DOWNLOAD_HANDLERS ----------------- -Default: ``{}`` - -A dict containing the request downloader handlers enabled in your project. -See `DOWNLOAD_HANDLERS_BASE` for example format. - -.. setting:: DOWNLOAD_HANDLERS_BASE - -DOWNLOAD_HANDLERS_BASE ----------------------- - Default:: { 'file': 'scrapy.core.downloader.handlers.file.FileDownloadHandler', - 'http': 'scrapy.core.downloader.handlers.http.HttpDownloadHandler', - 'https': 'scrapy.core.downloader.handlers.http.HttpDownloadHandler', + 'http': 'scrapy.core.downloader.handlers.http.HTTPDownloadHandler', + 'https': 'scrapy.core.downloader.handlers.http.HTTPDownloadHandler', 's3': 'scrapy.core.downloader.handlers.s3.S3DownloadHandler', + 'ftp': 'scrapy.core.downloader.handlers.ftp.FTPDownloadHandler', } -A dict containing the request download handlers enabled by default in Scrapy. -You should never modify this setting in your project, modify -:setting:`DOWNLOAD_HANDLERS` instead. -If you want to disable any of the above download handlers you must define them -in your project's :setting:`DOWNLOAD_HANDLERS` setting and assign `None` -as their value. For example, if you want to disable the file download -handler:: +A dict containing the request downloader handlers enabled in your project. + +When you set :setting:`DOWNLOAD_HANDLERS` manually, e.g. in your project's +settings module, it will be merged with the default, not overwrite it. If you +want to disable any of the default download handlers you must assign ``None`` +as their value. For example, if you want to disable the file download handler:: DOWNLOAD_HANDLERS = { 'file': None, @@ -552,15 +544,6 @@ to ``vi`` (on Unix systems) or the IDLE editor (on Windows). EXTENSIONS ---------- -Default:: ``{}`` - -A dict containing the extensions enabled in your project, and their orders. - -.. setting:: EXTENSIONS_BASE - -EXTENSIONS_BASE ---------------- - Default:: { @@ -575,13 +558,19 @@ Default:: 'scrapy.extensions.throttle.AutoThrottle': 0, } -The list of available extensions. Keep in mind that some of them need to -be enabled through a setting. By default, this setting contains all stable -built-in extensions. +A dict containing the extensions enabled in your project, and their orders. By +default, this setting contains all stable built-in extensions. Keep in mind that +some of them need to be enabled through a setting. + +When you set :setting:`EXTENSIONS` manually, e.g. in your project's settings +module, it will be merged with the default, not overwrite it. If you want to +disable any of the default enabled extensions you must assign ``None`` as their +value. For more information See the :ref:`extensions user guide ` and the :ref:`list of available extensions `. + .. setting:: ITEM_PIPELINES ITEM_PIPELINES @@ -589,12 +578,9 @@ ITEM_PIPELINES Default: ``{}`` -A dict containing the item pipelines to use, and their orders. The dict is -empty by default order values are arbitrary but it's customary to define them -in the 0-1000 range. - -Lists are supported in :setting:`ITEM_PIPELINES` for backwards compatibility, -but they are deprecated. +A dict containing the item pipelines to use, and their orders. Order values are +arbitrary, but it is customary to define them in the 0-1000 range. Lower orders +process before higher orders. Example:: @@ -603,16 +589,6 @@ Example:: 'mybot.pipelines.validate.StoreMyItem': 800, } -.. setting:: ITEM_PIPELINES_BASE - -ITEM_PIPELINES_BASE -------------------- - -Default: ``{}`` - -A dict containing the pipelines enabled by default in Scrapy. You should never -modify this setting in your project, modify :setting:`ITEM_PIPELINES` instead. - .. setting:: LOG_ENABLED LOG_ENABLED @@ -638,7 +614,7 @@ LOG_FILE Default: ``None`` -File name to use for logging output. If None, standard error will be used. +File name to use for logging output. If ``None``, standard error will be used. .. setting:: LOG_FORMAT @@ -902,16 +878,6 @@ The scheduler to use for crawling. SPIDER_CONTRACTS ---------------- -Default:: ``{}`` - -A dict containing the scrapy contracts enabled in your project, used for -testing spiders. For more info see :ref:`topics-contracts`. - -.. setting:: SPIDER_CONTRACTS_BASE - -SPIDER_CONTRACTS_BASE ---------------------- - Default:: { @@ -920,9 +886,13 @@ Default:: 'scrapy.contracts.default.ScrapesContract': 3, } -A dict containing the scrapy contracts enabled by default in Scrapy. You should -never modify this setting in your project, modify :setting:`SPIDER_CONTRACTS` -instead. For more info see :ref:`topics-contracts`. +A dict containing the scrapy contracts enabled in your project, used for +testing spiders. For more info see :ref:`topics-contracts`. + +When you set :setting:`SPIDER_CONTRACTS` manually, e.g. in your project's +settings module, it will be merged with the default, not overwrite it. If you +want to disable any of the default contracts you must assign ``None`` as their +value. .. setting:: SPIDER_LOADER_CLASS @@ -939,16 +909,6 @@ The class that will be used for loading spiders, which must implement the SPIDER_MIDDLEWARES ------------------ -Default:: ``{}`` - -A dict containing the spider middlewares enabled in your project, and their -orders. For more info see :ref:`topics-spider-middleware-setting`. - -.. setting:: SPIDER_MIDDLEWARES_BASE - -SPIDER_MIDDLEWARES_BASE ------------------------ - Default:: { @@ -959,10 +919,14 @@ Default:: 'scrapy.spidermiddlewares.depth.DepthMiddleware': 900, } -A dict containing the spider middlewares enabled by default in Scrapy. You -should never modify this setting in your project, modify -:setting:`SPIDER_MIDDLEWARES` instead. For more info see -:ref:`topics-spider-middleware-setting`. +A dict containing the spider middlewares enabled in your project, and their +orders. Low orders are closer to the engine, high orders are closer to the +spider. For more info see :ref:`topics-spider-middleware-setting`. + +When you set :setting:`SPIDER_MIDDLEWARES` manually, e.g. in your project's +settings module, it will be merged with the default, not overwrite it. If you +want to disable any of the default spider middlewares you must assign ``None`` +as their value. .. setting:: SPIDER_MODULES diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst index 84daaaa55..d448801d3 100644 --- a/docs/topics/spider-middleware.rst +++ b/docs/topics/spider-middleware.rst @@ -24,22 +24,20 @@ Here's an example:: 'myproject.middlewares.CustomSpiderMiddleware': 543, } -The :setting:`SPIDER_MIDDLEWARES` setting is merged with the -:setting:`SPIDER_MIDDLEWARES_BASE` setting defined in Scrapy (and not meant to -be overridden) and then sorted by order to get the final sorted list of enabled -middlewares: the first middleware is the one closer to the engine and the last -is the one closer to the spider. +The specified :setting:`SPIDER_MIDDLEWARES` setting is merged with the default +one (i.e. it does not overwrite it) and then sorted by order to get the final +sorted list of enabled middlewares: the first middleware is the one closer to +the engine and the last is the one closer to the spider. -To decide which order to assign to your middleware see the -:setting:`SPIDER_MIDDLEWARES_BASE` setting and pick a value according to where +To decide which order to assign to your middleware see the default +:setting:`SPIDER_MIDDLEWARES` setting and pick a value according to where you want to insert the middleware. The order does matter because each middleware performs a different action and your middleware could depend on some previous (or subsequent) middleware being applied. -If you want to disable a builtin middleware (the ones defined in -:setting:`SPIDER_MIDDLEWARES_BASE`, and enabled by default) you must define it -in your project :setting:`SPIDER_MIDDLEWARES` setting and assign `None` as its -value. For example, if you want to disable the off-site middleware:: +If you want to disable a builtin middleware you must define it in your project's +:setting:`SPIDER_MIDDLEWARES` setting and assign ``None`` as its value. For +example, if you want to disable the off-site middleware:: SPIDER_MIDDLEWARES = { 'myproject.middlewares.CustomSpiderMiddleware': 543, @@ -173,7 +171,7 @@ information on how to use them and how to write your own spider middleware, see the :ref:`spider middleware usage guide `. For a list of the components enabled by default (and their orders) see the -:setting:`SPIDER_MIDDLEWARES_BASE` setting. +:setting:`SPIDER_MIDDLEWARES` setting. DepthMiddleware --------------- diff --git a/scrapy/commands/check.py b/scrapy/commands/check.py index 017595f04..a423ba2c9 100644 --- a/scrapy/commands/check.py +++ b/scrapy/commands/check.py @@ -58,10 +58,7 @@ class Command(ScrapyCommand): def run(self, args, opts): # load contracts - contracts = build_component_list( - self.settings['SPIDER_CONTRACTS_BASE'], - self.settings['SPIDER_CONTRACTS'], - ) + contracts = build_component_list(self.settings._getcomposite('SPIDER_CONTRACTS')) conman = ContractsManager(load_object(c) for c in contracts) runner = TextTestRunner(verbosity=2 if opts.verbose else 1) result = TextTestResult(runner.stream, runner.descriptions, runner.verbosity) diff --git a/scrapy/commands/crawl.py b/scrapy/commands/crawl.py index 72df11476..9c8a3d4ce 100644 --- a/scrapy/commands/crawl.py +++ b/scrapy/commands/crawl.py @@ -1,6 +1,6 @@ import os from scrapy.commands import ScrapyCommand -from scrapy.utils.conf import arglist_to_dict +from scrapy.utils.conf import arglist_to_dict, remove_none_values from scrapy.exceptions import UsageError @@ -34,10 +34,8 @@ class Command(ScrapyCommand): self.settings.set('FEED_URI', 'stdout:', priority='cmdline') else: self.settings.set('FEED_URI', opts.output, priority='cmdline') - valid_output_formats = ( - list(self.settings.getdict('FEED_EXPORTERS').keys()) + - list(self.settings.getdict('FEED_EXPORTERS_BASE').keys()) - ) + feed_exporters = remove_none_values(self.settings._getcomposite('FEED_EXPORTERS')) + valid_output_formats = feed_exporters.keys() if not opts.output_format: opts.output_format = os.path.splitext(opts.output)[1].replace(".", "") if opts.output_format not in valid_output_formats: diff --git a/scrapy/commands/runspider.py b/scrapy/commands/runspider.py index 88f5a3015..7d85984c3 100644 --- a/scrapy/commands/runspider.py +++ b/scrapy/commands/runspider.py @@ -5,7 +5,7 @@ from importlib import import_module from scrapy.utils.spider import iter_spider_classes from scrapy.commands import ScrapyCommand from scrapy.exceptions import UsageError -from scrapy.utils.conf import arglist_to_dict +from scrapy.utils.conf import arglist_to_dict, remove_none_values def _import_file(filepath): @@ -57,10 +57,8 @@ class Command(ScrapyCommand): self.settings.set('FEED_URI', 'stdout:', priority='cmdline') else: self.settings.set('FEED_URI', opts.output, priority='cmdline') - valid_output_formats = ( - list(self.settings.getdict('FEED_EXPORTERS').keys()) + - list(self.settings.getdict('FEED_EXPORTERS_BASE').keys()) - ) + feed_exporters = remove_none_values(self.settings._getcomposite('FEED_EXPORTERS')) + valid_output_formats = feed_exporters.keys() if not opts.output_format: opts.output_format = os.path.splitext(opts.output)[1].replace(".", "") if opts.output_format not in valid_output_formats: diff --git a/scrapy/core/downloader/handlers/__init__.py b/scrapy/core/downloader/handlers/__init__.py index 6c9514af6..9b118c39b 100644 --- a/scrapy/core/downloader/handlers/__init__.py +++ b/scrapy/core/downloader/handlers/__init__.py @@ -4,6 +4,7 @@ import logging from twisted.internet import defer import six from scrapy.exceptions import NotSupported, NotConfigured +from scrapy.utils.conf import remove_none_values from scrapy.utils.httpobj import urlparse_cached from scrapy.utils.misc import load_object from scrapy import signals @@ -19,13 +20,8 @@ class DownloadHandlers(object): self._schemes = {} # stores acceptable schemes on instancing self._handlers = {} # stores instanced handlers for schemes self._notconfigured = {} # remembers failed handlers - handlers = crawler.settings.get('DOWNLOAD_HANDLERS_BASE') - handlers.update(crawler.settings.get('DOWNLOAD_HANDLERS', {})) + handlers = remove_none_values(crawler.settings._getcomposite('DOWNLOAD_HANDLERS')) for scheme, clspath in six.iteritems(handlers): - # Allow to disable a handler just like any other - # component (extension, middleware, etc). - if clspath is None: - continue self._schemes[scheme] = clspath crawler.signals.connect(self._close, signals.engine_stopped) diff --git a/scrapy/core/downloader/middleware.py b/scrapy/core/downloader/middleware.py index 9cd30c144..958113fc3 100644 --- a/scrapy/core/downloader/middleware.py +++ b/scrapy/core/downloader/middleware.py @@ -19,8 +19,7 @@ class DownloaderMiddlewareManager(MiddlewareManager): @classmethod def _get_mwlist_from_settings(cls, settings): - return build_component_list(settings['DOWNLOADER_MIDDLEWARES_BASE'], \ - settings['DOWNLOADER_MIDDLEWARES']) + return build_component_list(settings._getcomposite('DOWNLOADER_MIDDLEWARES')) def _add_middleware(self, mw): if hasattr(mw, 'process_request'): diff --git a/scrapy/core/spidermw.py b/scrapy/core/spidermw.py index c1c5b10fc..b5c80c350 100644 --- a/scrapy/core/spidermw.py +++ b/scrapy/core/spidermw.py @@ -18,8 +18,7 @@ class SpiderMiddlewareManager(MiddlewareManager): @classmethod def _get_mwlist_from_settings(cls, settings): - return build_component_list(settings['SPIDER_MIDDLEWARES_BASE'], \ - settings['SPIDER_MIDDLEWARES']) + return build_component_list(settings._getcomposite('SPIDER_MIDDLEWARES')) def _add_middleware(self, mw): super(SpiderMiddlewareManager, self)._add_middleware(mw) diff --git a/scrapy/downloadermiddlewares/defaultheaders.py b/scrapy/downloadermiddlewares/defaultheaders.py index f1d2bd631..c8924c04a 100644 --- a/scrapy/downloadermiddlewares/defaultheaders.py +++ b/scrapy/downloadermiddlewares/defaultheaders.py @@ -4,6 +4,8 @@ DefaultHeaders downloader middleware See documentation in docs/topics/downloader-middleware.rst """ +from scrapy.utils.conf import remove_none_values + class DefaultHeadersMiddleware(object): @@ -12,7 +14,8 @@ class DefaultHeadersMiddleware(object): @classmethod def from_crawler(cls, crawler): - return cls(crawler.settings.get('DEFAULT_REQUEST_HEADERS').items()) + headers = remove_none_values(crawler.settings['DEFAULT_REQUEST_HEADERS']) + return cls(headers.items()) def process_request(self, request, spider): for k, v in self._headers: diff --git a/scrapy/extension.py b/scrapy/extension.py index f68b1ba68..4ceb32c68 100644 --- a/scrapy/extension.py +++ b/scrapy/extension.py @@ -12,5 +12,4 @@ class ExtensionManager(MiddlewareManager): @classmethod def _get_mwlist_from_settings(cls, settings): - return build_component_list(settings['EXTENSIONS_BASE'], \ - settings['EXTENSIONS']) + return build_component_list(settings._getcomposite('EXTENSIONS')) diff --git a/scrapy/extensions/feedexport.py b/scrapy/extensions/feedexport.py index 7560e89d3..fb07657d6 100644 --- a/scrapy/extensions/feedexport.py +++ b/scrapy/extensions/feedexport.py @@ -18,6 +18,7 @@ from twisted.internet import defer, threads from w3lib.url import file_uri_to_path from scrapy import signals +from scrapy.utils.conf import remove_none_values from scrapy.utils.ftp import ftp_makedirs_cwd from scrapy.exceptions import NotConfigured from scrapy.utils.misc import load_object @@ -195,8 +196,7 @@ class FeedExporter(object): return item def _load_components(self, setting_prefix): - conf = dict(self.settings['%s_BASE' % setting_prefix]) - conf.update(self.settings[setting_prefix]) + conf = remove_none_values(self.settings._getcomposite(setting_prefix)) d = {} for k, v in conf.items(): try: diff --git a/scrapy/pipelines/__init__.py b/scrapy/pipelines/__init__.py index d433498f5..8df0d3154 100644 --- a/scrapy/pipelines/__init__.py +++ b/scrapy/pipelines/__init__.py @@ -13,15 +13,7 @@ class ItemPipelineManager(MiddlewareManager): @classmethod def _get_mwlist_from_settings(cls, settings): - item_pipelines = settings['ITEM_PIPELINES'] - if isinstance(item_pipelines, (tuple, list, set, frozenset)): - from scrapy.exceptions import ScrapyDeprecationWarning - import warnings - warnings.warn('ITEM_PIPELINES defined as a list or a set is deprecated, switch to a dict', - category=ScrapyDeprecationWarning, stacklevel=1) - # convert old ITEM_PIPELINE list to a dict with order 500 - item_pipelines = dict(zip(item_pipelines, range(500, 500+len(item_pipelines)))) - return build_component_list(settings['ITEM_PIPELINES_BASE'], item_pipelines) + return build_component_list(settings._getcomposite('ITEM_PIPELINES')) def _add_middleware(self, pipe): super(ItemPipelineManager, self)._add_middleware(pipe) diff --git a/scrapy/settings/__init__.py b/scrapy/settings/__init__.py index fa7fa3178..7eea562e1 100644 --- a/scrapy/settings/__init__.py +++ b/scrapy/settings/__init__.py @@ -36,13 +36,21 @@ class SettingsAttribute(object): def __init__(self, value, priority): self.value = value - self.priority = priority + if isinstance(self.value, BaseSettings): + self.priority = max(self.value.maxpriority(), priority) + else: + self.priority = priority def set(self, value, priority): """Sets value if priority is higher or equal than current priority.""" - if priority >= self.priority: - self.value = value - self.priority = priority + if isinstance(self.value, BaseSettings): + # Ignore self.priority if self.value has per-key priorities + self.value.update(value, priority) + self.priority = max(self.value.maxpriority(), priority) + else: + if priority >= self.priority: + self.value = value + self.priority = priority def __str__(self): return "