RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
1,895 Commits 49 Branches 132 Tags 137 MiB
Commit Graph

1895 Commits

Author SHA1 Message Date
Pablo Hoffman 024c00a512 title-cased project name in setup.py 2009-12-12 15:48:02 -02:00
Pablo Hoffman a953efd8e5 Automated merge with http://hg.scrapy.org/scrapy-stable 2009-12-12 15:40:16 -02:00
Pablo Hoffman f22fe95ae9 changed version to 0.8-rc1 2009-12-12 15:31:03 -02:00
Pablo Hoffman 3d6f598abc added minor delay to httpcache tests, for them to pass on win32 2009-12-12 15:30:21 -02:00
Pablo Hoffman 9c31fc447a Changed character used in unit test because it's not a valid file name char on win32 2009-12-12 11:07:50 -02:00
Pablo Hoffman 3012030b2f Fixed bug in file:// downloader handler with uris containing percent-escaped chars 2009-12-12 10:57:17 -02:00
Pablo Hoffman cca4be4d64 uncommented some lines in extras/build_release.sh 2009-12-11 23:52:53 -02:00
Ismael Carnales 4ecc909bc1 Fix RobotsTxtMiddleware reference in doc 2009-12-04 15:37:24 -02:00
Pablo Hoffman fe88aad27d removed some old code from scrapy.spider.models 2009-12-02 23:30:00 -02:00
Pablo Hoffman 41894c776e adjusted some core code and comments for consistency after the move to spider references 2009-12-02 23:06:17 -02:00
Pablo Hoffman 57bae5335f Added Settings class tests, and fixed minor bug 2009-12-02 15:52:17 -02:00
Pablo Hoffman dbe0a6875c Decoupled Settings class from environment settings singleton (scrapy.conf.settings) 2009-12-02 15:11:11 -02:00
Pablo Hoffman 925eecce70 fixed bug in OpenSSL depedency test which was comparing versions alphabetically, instead of numerically 2009-12-02 13:59:58 -02:00
Pablo Hoffman dda7e01d2f don't fail if OpenSSL is not installed in test_dependencies.py 2009-12-02 11:26:34 -02:00
Ismael Carnales 07344666e2 Move webconsole extensions doc to webconsole topic 2009-12-01 10:47:11 -02:00
Ismael Carnales e694c8ed02 Remove domain references in close spider extension doc 2009-11-30 11:38:56 -02:00
Ismael Carnales 12a7ff7312 Rename Close domain to close spider in extensions doc 2009-11-30 11:36:18 -02:00
Ismael Carnales 8d9cedd88b Reorder signals doc to respect alphabetical order 2009-11-30 11:29:19 -02:00
Ismael Carnales 93cc3d2715 Correct param formatting in item pipelines doc 2009-11-30 11:04:15 -02:00
Pablo Hoffman cabee59d4d Fixed compatibility issues with Twisted 9 (closes #128) 2009-11-28 22:15:09 -02:00
Pablo Hoffman 6084be3b2e added iter_all() function to scrapy.util.trackref module and improved memory leaks documentation. also added a new FAQ antry about memory issues 2009-11-28 16:21:59 -02:00
Pablo Hoffman de35eee3cd Automated merge after applying ajones patch 2009-11-28 12:31:05 -02:00
Pablo Hoffman 207aae2b04 Fixed logging of "Spider closed" message in engine 2009-11-26 19:12:33 -02:00
Pablo Hoffman 34fcf6ba5b Added informative message when trying to use trackref and it's not enabled 2009-11-26 18:37:45 -02:00
Pablo Hoffman fd50891113 Keep track of pending next-request calls in the engine and downloader, to cancel them properly when the spider is closed. This also avoids keeping spider references alive, after they are closed. 2009-11-26 17:00:16 -02:00
Pablo Hoffman c0f1c8de04 use DelayedCall.active() instead DelayedCall.called, to support cancelled tasks 2009-11-26 16:10:28 -02:00
Pablo Hoffman 88417a3ed1 Fixed bug in LiveStats webconsole module which was keeping references to spiders alive, after they were closed 2009-11-25 22:51:58 -02:00
Pablo Hoffman a2854e3948 Added hack to speed up processing of IgnoreRequest errors (#125) 2009-11-25 22:28:59 -02:00
Pablo Hoffman a48516b105 removed obsolete remove_escape_chars function - use replace_escape_chars instead 2009-11-25 22:09:47 -02:00
ajones1@gmail.com 0dcf47a706 fix name errors on robots.txt middleware during spider_close 2009-11-25 12:59:28 -08:00
Pablo Hoffman a36909a3f9 Added BaseSpider objects to trackref 2009-11-25 16:59:47 -02:00
Pablo Hoffman 9311741236 Fixed memory leak in CachingResolver 2009-11-25 16:54:36 -02:00
Pablo Hoffman 1cfd959828 remove wrong support for returning Responses in scheduler middlewares 2009-11-25 11:20:43 -02:00
Pablo Hoffman 1a81ab6f07 renamed extension: DelayedCloseDomain to SpiderCloseDelay 
--HG--
rename : scrapy/contrib/delayedclosedomain.py => scrapy/contrib/spiderclosedelay.py
 2009-11-21 15:17:38 -02:00
Pablo Hoffman 326090d4a4 Reordered setting to preseve alphabetical order 2009-11-21 15:16:29 -02:00
Pablo Hoffman a49aef2beb Renamed exception: DontCloseDomain to DontCloseSpider (closes #120) 2009-11-21 15:06:03 -02:00
Pablo Hoffman 0d75a3a636 Automated merge with http://hg.scrapy.org/scrapy-stable/ 2009-11-20 10:38:51 -02:00
Ismael Carnales f86f62c5d9 Use setuptools for install, if not present fallback to distutils 2009-11-20 09:30:06 -02:00
Pablo Hoffman dd662e09d8 some minor fixes to scheduler middleware doc 2009-11-19 12:23:54 -02:00
Pablo Hoffman 8c14abadbb Automated merge with http://hg.scrapy.org/scrapy-stable/ 2009-11-19 11:23:50 -02:00
Pablo Hoffman 9ff4dbf636 renamed file missing from previous commit 
--HG--
rename : scrapy/xlib/patches.py => scrapy/xlib/twisted_250_monkeypatches.py
 2009-11-19 11:23:36 -02:00
Pablo Hoffman f4e93700bd Automated merge with http://hg.scrapy.org/scrapy-stable/ 2009-11-19 10:44:02 -02:00
Pablo Hoffman bf55a4708f prevent 'import scrapy' from failing when twisted module is not available, also moved twisted 2.5.0 monkeypatch into a more specific module name 2009-11-19 10:41:36 -02:00
Pablo Hoffman c4f77c4da0 minor fixes to images doc (thanks amccloud) 2009-11-16 11:15:25 -02:00
Pablo Hoffman 0d6aee1f12 updated wrong documentation 2009-11-13 20:03:56 -02:00
Daniel Grana 445d8cd9f0 delay next_request check after stoping a spider close to avoid 100% cpu usage loops in some cases 2009-11-06 22:02:12 -02:00
Pablo Hoffman aeab5370cb StatsCollector: ported methods to receive spider instances (closes #113), removed list_domains() method, added iter_spider_stats() method 2009-11-14 20:28:59 -02:00
Pablo Hoffman c4c6e7c8cd Automated merge with http://hg.scrapy.org/scrapy-stable/ 2009-11-13 20:04:39 -02:00
Pablo Hoffman 505dfe6f7e fixed exception when running scrapy shell on a non-textual response (fixes #116) 2009-11-13 17:21:59 -02:00
Pablo Hoffman f3e861c89e replaced old reference to domain instead of spider 2009-11-13 14:44:03 -02:00