Since python 2.7, importlib.import_module is the recommended way to
import modules programmatically.
From __import__'s doc:
Import a module. Because this function is meant for use by the
Python
interpreter and not for general use it is better to use
importlib.import_module() to programmatically import a module.
Previously, it failed silently if an ImportError was caught when trying
to import the scrapy settings module. This not only happened when the
scrapy settings module itself was missing, but also when it tried to
import a missing module, which made the whole thing a bad idea.
A side effect of this change (not required, but for simplification) is
that we no longer support the default "scrapy_settings" name for the
scrapy settings module, but this was never used afaik.
/usr/local/lib/python2.7/dist-packages/Scrapy-0.15.1-py2.7.egg/scrapy/utils/project.py:17: UserWarning: Cannot import scrapy settings module settings
warnings.warn("Cannot import scrapy settings module %s" % scrapy_module)
...
File "/usr/local/lib/python2.7/dist-packages/Scrapy-0.15.1-py2.7.egg/scrapy/cmdline.py", line 117, in _run_command
cmd.run(args, opts)
File "/usr/local/lib/python2.7/dist-packages/Scrapy-0.15.1-py2.7.egg/scrapy/commands/crawl.py", line 43, in run
spider = self.crawler.spiders.create(spname, **opts.spargs)
File "/usr/local/lib/python2.7/dist-packages/Scrapy-0.15.1-py2.7.egg/scrapy/spidermanager.py", line 43, in create
raise KeyError("Spider not found: %s" % spider_name)
KeyError: 'Spider not found: fb_spider'
Which is not very descriptive.
Now showing details about the exception when ImportError was raises
* improved detection of inside-project environments
* make list command faster (by only instantiating the spider manger)
* print a warning when extensions (middlewares, etc) are disabled with a message on NotConfigured exception
* assert that scrapy configuration hasn't been loaded in scrapyd.runner
* simplified IgnoreRequest exception, to avoid loading settings when importing scrapy.exceptions
* added test to make sure certain modules don't cause scrapy.conf module to be
loaded, to ensure the scrapyd runner bootstraping performs properly
* made it use the project data storage by default (closes#279)
* added HTTPCACHE_ENABLED setting (False by default) to enable it
* made HTTPCACHE_DIR = 'httpcache' by default (inside the project data storage)
* simplified HTTPCACHE_EXPIRATION_SECS semantics: zero means don't expire,
dropped support for negative numbers
* other minor doc improvements
Rolando Espinoza La fuente
Pablo Hoffman
Victor Varvariuc
Shane Evans