* Cover 2.15.0 in the release notes (up to 31bf7c3).
* Replace VERSION.
* Document unified download handler exceptions.
* Updates.
* Cover 2.15.0 in the release notes (up to 0c6ccf5).
* Cover 2.15.0 in the release notes (up to b68f267).
* Run linkcheck and fix issues.
* Cleanup unused link targets.
* Cleanup.
* Use AsyncCrawlerProcess if TWISTED_REACTOR_ENABLED=False.
* defer.Deferred -> Deferred.
* Allow Shell with TWISTED_REACTOR_ENABLED=False.
* Shell workflow notes.
* Remove an unused argument.
* Shell.fetch_available.
* Shell._use_reactor.
* Some more comments.
* Simplify _schedule() and fix the spider shell var.
* More async def.
* Reactorless shell support.
* Add pragma: no cover.
* More notes.
* Remove a TODO.
* Remove the chdir fixture.
* Re-enable fancy pytest asserts.
* Remove doc files from pytest ignores.
* Restore docs/_ext in test collection ignores.
* Skip a doctest that fails on Windows.
* Fix tests that were writing to the current dir.
Change python code snippets to begin with '.. code-block:: python' to be recognized by the hook for formatting. All snippets under '::' (rst literal blocks) are ignored.
* update the scrapy.org example to deal with the new layout.
* replaced slashdot.org by reddit.com, because it seems that slashdot is blocking requests.
I've made a few small corrections, some spelling changes and typo fixes.
I've tried to respect regional spelling differences and avoided proposing hyphenating compound words.
Please enter the commit message for your changes. Lines starting
Adds support for configuration of shells from scrapy.cfg
and SCRAPY_PYTHON_SHELL.
config snippet:
cat <<EOF >> ~/.scrapy.cfg
[settings]
# shell can be one of ipython, bpython or python;
# to be tried as the interactive python console
# (in above order, unless set here).
shell = python
EOF
(closes#270, #1100, #1301)
The current documentation has references to the deprecated `sel` when interacting with the shell. I've removed them and replaced uses of `sel.xpath` with `response.xpath` instead.
* Rename BaseSpider.domain_name to BaseSpider.name
This patch implements the domain_name to name change in BaseSpider class and
change all spider instantiations to use the new attribute.
* Add allowed_domains to spider
This patch implements the merging of spider.domain_name and
spider.extra_domain_names in spider.allowed_domains for offsite checking
purposes.
Note that spider.domain_name is not touched by this patch, only not used.
* Remove spider.domain_name references from scrapy.stats
* Rename domain_stats to spider_stats in MemoryStatsCollector
* Use ``spider`` instead of ``domain`` in SimpledbStatsCollector
* Rename domain_stats_history table to spider_data_history and rename domain
field to spider in MysqlStatsCollector
* Refactor genspider command
The new signature for genspider is: genspider [options] <domain_name>.
Genspider uses domain_name for spider name and for the module name.
* Remove spider.domain_name references
* Update crawl command signature <spider|url>
* docs: updated references to domain_name
* examples/experimental: use spider.name
* genspider: require <name> <domain>
* spidermanager: renamed crawl_domain to crawl_spider_name
* spiderctl: updated references of *domain* to spider
* added backward compatiblity with legacy spider's attributes
'domain_name' and 'extra_domain_names'
Andrey Rakhmatullin
Sebastien Dionne
pankaj1707k
Kian-Meng, Ang
Marlena Chatzigrigoriou
Adrián Chaves
sakshamb2113
nyov
Mikhail Korobov
Marc Hernandez Cabot
Aditya
Elias Dorneles
Paul Tremberth
Valdir Stumm Jr
smirecki
Julia Medina
Diana Huang
Lisimia
Rolando Espinoza
Daniel Graña
Pablo Hoffman