9ee92686c7
Moved spidermanager tests module according to policies
...
--HG--
rename : scrapy/tests/test_contrib_spidermanager/__init__.py => scrapy/tests/test_spidermanager/__init__.py
rename : scrapy/tests/test_contrib_spidermanager/test_spiders/__init__.py => scrapy/tests/test_spidermanager/test_spiders/__init__.py
rename : scrapy/tests/test_contrib_spidermanager/test_spiders/spider0.py => scrapy/tests/test_spidermanager/test_spiders/spider0.py
rename : scrapy/tests/test_contrib_spidermanager/test_spiders/spider1.py => scrapy/tests/test_spidermanager/test_spiders/spider1.py
rename : scrapy/tests/test_contrib_spidermanager/test_spiders/spider2.py => scrapy/tests/test_spidermanager/test_spiders/spider2.py
rename : scrapy/tests/test_contrib_spidermanager/test_spiders/spider3.py => scrapy/tests/test_spidermanager/test_spiders/spider3.py
2010-08-23 00:25:03 -03:00
58b4cc2c32
Some minor fixes to contribution Contributing documentation
2010-08-23 00:23:14 -03:00
6585c1a28f
removed (somewhat hacky) MAIL_DEBUG setting
2010-08-22 22:42:00 -03:00
e189861b46
Fixed Item Loader bug that was preventing values that evaluate to False from being loaded. Patch contributed by Anibal Pacheco. Closes #174
2010-08-22 22:07:44 -03:00
254d517da1
Moved module: scrapy.contrib.spidermanager to scrapy.spidermanager
...
--HG--
rename : scrapy/contrib/spidermanager.py => scrapy/spidermanager.py
2010-08-22 21:53:47 -03:00
cf8a085f44
Minor improvement to bash autocompletion
2010-08-22 20:10:11 -03:00
fd784cd131
Fixed tests on Windows
2010-08-22 19:37:20 -03:00
33686fa563
better skip test message
2010-08-22 19:08:52 -03:00
6da1162839
minor fixes to FAQ
2010-08-22 19:08:45 -03:00
b3753d34eb
Added FAQ entry about feed exports
2010-08-22 05:59:30 -03:00
cbfec4bb0e
Renamed webservice ManagerResource to CrawlerResource
...
--HG--
rename : scrapy/contrib/webservice/manager.py => scrapy/contrib/webservice/crawler.py
2010-08-22 05:48:03 -03:00
7546a0805c
Removed webservice Spiders and Extensions resources since they can now be accessed through the Execution Manager (aka. Crawler) resource
2010-08-22 05:38:46 -03:00
e5474d8cc6
Made ExtensionManager a subclass of MiddlewareManager
2010-08-22 05:33:08 -03:00
c1225e0f45
"parse" command refactoring. This fixes #173 and renders #106 invalid.
2010-08-22 05:04:17 -03:00
9fccc11363
Moved scrapy.extension.extensions singleton to a "extensions" attribute of the scrapy.project.crawler singleton. Refs #189
2010-08-22 02:15:11 -03:00
52c1e137e5
Moved scrapy.spider.spiders singleton to a "spiders" attribute of the scrapy.project.crawler singleton. Refs #189
...
Warning: this is a backwards incompatible change.
--HG--
rename : scrapy/spider/models.py => scrapy/spider.py
2010-08-22 02:15:10 -03:00
faf7a7da83
Moved scrapymanager singleton to scrapy.project module. Refs #189
...
Detail of changes:
* Moved scrapy.core.manager.ExecutionManager class to scrapy.crawler.Crawler
* Added scrapy.project.crawler singleton to reference a singleton instance of
Crawler class (previously known as scrapymanager)
* Left an alias scrapy.core.manager.scrapymanager to scrapy.project.crawler for
backwards compatibility (to be removed in Scrapy 0.11)
2010-08-22 02:10:53 -03:00
053d45e79f
Splitted stats collector classes from stats collection facility ( #204 )
...
* moved scrapy.stats.collector.__init__ module to scrapy.statscol
* moved scrapy.stats.collector.simpledb module to scrapy.contrib.statscol
* moved signals from scrapy.stats.signals to scrapy.signals
* moved scrapy/stats/__init__.py to scrapy/stats.py
* updated documentation and tests accordingly
--HG--
rename : scrapy/stats/collector/simpledb.py => scrapy/contrib/statscol.py
rename : scrapy/stats/__init__.py => scrapy/stats.py
rename : scrapy/stats/collector/__init__.py => scrapy/statscol.py
2010-08-22 01:24:07 -03:00
c276c48c91
Added settings to Scrapy shell variables
2010-08-21 05:10:06 -03:00
b8b7b5ad74
Improved some commands descriptions
2010-08-21 05:03:38 -03:00
e6d2a3087e
example projects: added scrapy.cfg and removed scrapy-ctl.py
2010-08-21 04:56:19 -03:00
68f9fcffe8
genspider command refactoring. Also updated tests and doc
2010-08-21 04:46:48 -03:00
0da6132136
Made command-line too output more concise
2010-08-21 03:37:59 -03:00
0d9e75c684
Added bash completion for the Scrapy command-line tool. Closes #210
2010-08-21 03:23:45 -03:00
f80ae9af66
Fixed missing reference to old 'start' command. Refs #209
2010-08-21 01:44:08 -03:00
50621b7ef3
Renamed command "start" to "runserver". Closes #209
...
--HG--
rename : scrapy/commands/start.py => scrapy/commands/runserver.py
2010-08-21 01:42:55 -03:00
9aefa242d5
Applied documentation patch provided by Lucian Ursu ( closes #207 )
2010-08-21 01:26:35 -03:00
f782245c5a
Removed obsolete files
2010-08-21 01:24:39 -03:00
1d3b9e2ca8
Scrapy shell refactoring
2010-08-20 11:26:14 -03:00
7858244dca
Scrapy shell: moved python console starting code to scrapy.utils.console and get rid of noisy console banners
2010-08-20 01:33:02 -03:00
136b0e7450
Minor change to log message
2010-08-19 21:14:27 -03:00
6dd76ab54b
Fixed bug in Scrapy shell which hanged if requests failed to download ( #205 ), added dont_filter=True to requests generated when calling the shell with a url argument, and changed formatting of messages
2010-08-19 21:11:39 -03:00
30e2404d8f
updated FAQ entry to recommend using higher download delays
2010-08-19 17:59:52 -03:00
66525e77c7
Improved support for scrapy-ctl -> scrapy migration by generating the scrapy.cfg file automatically if it doensn't exist
2010-08-19 17:57:34 -03:00
3d8151bb26
Added FAQ entry about response code 999
2010-08-19 16:51:51 -03:00
2ff5a83b7a
Added persistent execution queue (based on SQLite), and a new 'queue' command to control it. Closes #198
2010-08-19 02:55:52 -03:00
9740ad62a6
Added tests for runspider command
2010-08-19 02:30:15 -03:00
abd7a5e221
Fixed bug with runspider command that appeared after the introduction of new spider manager in r2121. Also factored out common code shared by new spider manager and runspider command, with tests included.
2010-08-19 01:58:10 -03:00
85f3a4a603
removed obsolete setting
2010-08-19 00:07:14 -03:00
94ead94bf6
Improved documentation of Scrapy command-line tool
...
--HG--
rename : docs/topics/cmdline.rst => docs/topics/commands.rst
2010-08-19 00:04:52 -03:00
34554da201
Deprecated scrapy-ctl.py command in favour of simpler "scrapy" command. Closes #199 . Also updated documenation accordingly and added convenient scrapy.bat script for running from Windows.
...
--HG--
rename : debian/scrapy-ctl.1 => debian/scrapy.1
rename : docs/topics/scrapy-ctl.rst => docs/topics/cmdline.rst
2010-08-18 19:48:32 -03:00
5522afca8d
pipeline process_item methods decorated with inlineCallbacks fails because of backwards compatible change to support inverted arguments
2010-08-18 13:05:50 -03:00
50f2236d04
removed hacky command_executed signal
2010-08-17 18:31:16 -03:00
a71521bfba
Default per-command settings are now specified in the default_settings attribute of the command object. Closes #201
2010-08-17 18:30:13 -03:00
76e19baa29
minor setting fix
2010-08-17 14:48:38 -03:00
ad3fd0afe8
fixed minor formatting issue with new feed exports doc
2010-08-17 14:37:59 -03:00
e741a807d2
Added new Feed exports extension with documentation and storage tests. Closes #197 .
...
Also deprecated File export pipeline (to be removed in Scrapy 0.11).
Still need to add tests for FeedExport main extension code.
2010-08-17 14:27:48 -03:00
d17695ee4b
Added "scrapy" command with project settings auto-discovery. Refs #199
2010-08-17 00:59:24 -03:00
b563e56363
fixed bug in url_is_from_spider() when no allowed_domains class attribute is present
2010-08-16 12:25:51 -03:00
a28cf29008
Simplified BaseSpider code by removing backwards compatibility code
2010-08-16 10:10:57 -03:00
Pablo Hoffman
Daniel Grana