* Update project.py removed one 'hack', seems irrelevant.
As mentioned by @Gallaecio in issue #3871, the 'hack' is cleared. I also double checked whether the environment variable "SCRAPY_PICKLED_SETTINGS_TO_OVERRIDE" was ever set in our codebase and it turns out we didn't set it or used it anywhere else.So I guess the 'hack' was not used in the current version. Also the name of this environment variable rather doesn't suggest it was a boolean(it is used in an 'if' condition which has perplexed me )
* Update project.py
* Update project.py
How about this?
* Update project.py
* Update project.py
* Update scrapy/utils/project.py
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
* Update scrapy/utils/project.py
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
* Update project.py
Since python 2.7, importlib.import_module is the recommended way to
import modules programmatically.
From __import__'s doc:
Import a module. Because this function is meant for use by the
Python
interpreter and not for general use it is better to use
importlib.import_module() to programmatically import a module.
Previously, it failed silently if an ImportError was caught when trying
to import the scrapy settings module. This not only happened when the
scrapy settings module itself was missing, but also when it tried to
import a missing module, which made the whole thing a bad idea.
A side effect of this change (not required, but for simplification) is
that we no longer support the default "scrapy_settings" name for the
scrapy settings module, but this was never used afaik.
/usr/local/lib/python2.7/dist-packages/Scrapy-0.15.1-py2.7.egg/scrapy/utils/project.py:17: UserWarning: Cannot import scrapy settings module settings
warnings.warn("Cannot import scrapy settings module %s" % scrapy_module)
...
File "/usr/local/lib/python2.7/dist-packages/Scrapy-0.15.1-py2.7.egg/scrapy/cmdline.py", line 117, in _run_command
cmd.run(args, opts)
File "/usr/local/lib/python2.7/dist-packages/Scrapy-0.15.1-py2.7.egg/scrapy/commands/crawl.py", line 43, in run
spider = self.crawler.spiders.create(spname, **opts.spargs)
File "/usr/local/lib/python2.7/dist-packages/Scrapy-0.15.1-py2.7.egg/scrapy/spidermanager.py", line 43, in create
raise KeyError("Spider not found: %s" % spider_name)
KeyError: 'Spider not found: fb_spider'
Which is not very descriptive.
Now showing details about the exception when ImportError was raises
* improved detection of inside-project environments
* make list command faster (by only instantiating the spider manger)
* print a warning when extensions (middlewares, etc) are disabled with a message on NotConfigured exception
* assert that scrapy configuration hasn't been loaded in scrapyd.runner
* simplified IgnoreRequest exception, to avoid loading settings when importing scrapy.exceptions
* added test to make sure certain modules don't cause scrapy.conf module to be
loaded, to ensure the scrapyd runner bootstraping performs properly
* made it use the project data storage by default (closes#279)
* added HTTPCACHE_ENABLED setting (False by default) to enable it
* made HTTPCACHE_DIR = 'httpcache' by default (inside the project data storage)
* simplified HTTPCACHE_EXPIRATION_SECS semantics: zero means don't expire,
dropped support for negative numbers
* other minor doc improvements
Andrey Rakhmatullin
Vsevolod Breus
cakemd
Anderson Carlos Ferreira da Silva
Jalil SA
Emmanuel Rondan
Laerte Pereira
Matthew Donoughe
Eugenio Lacuesta
Ammar Najjar
Adrián Chaves
Drew Seibert
Shivam Sandbhor
Elias Dorneles
Felix Yan
Julia Medina
Rolando Espinoza La fuente
Pablo Hoffman
Victor Varvariuc
Shane Evans