RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
2,401 Commits 49 Branches 132 Tags 138 MiB
Commit Graph

2401 Commits

Author SHA1 Message Date
Pablo Hoffman f8b4d1dc5d Fixed compatibility with Python 2.5 2010-10-21 12:53:40 -02:00
Pablo Hoffman 6c921896a5 Expanded documentation on deploy command and versions. Refs #261 2010-10-19 00:11:45 -02:00
Pablo Hoffman 1d567cdce6 Added new 'deploy' command. Closes #261 2010-10-18 22:38:46 -02:00
Pablo Hoffman 7d8f922df9 Added documentation for CLOSESPIDER_ERRORCOUNT setting. Refs #254 2010-10-18 22:36:30 -02:00
Pablo Hoffman c96f17c43d Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-10-18 03:21:21 -02:00
Pablo Hoffman deb9c7ef04 Reversed scrapy.cfg lookup order so that the one in the current project has more precedence. Also added alternative system-wide location for windows. 2010-10-18 03:18:54 -02:00
Pablo Hoffman 98662e53ea Formatting fix in Scrapyd doc 2010-10-17 03:20:23 -02:00
Pablo Hoffman a3d85da96f Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-10-16 19:54:24 -02:00
Pablo Hoffman 5f65c26080 Some minor improvements to feature list in Scrapy at a Glance documentation page 2010-10-16 19:02:08 -02:00
Pablo Hoffman 9d16ff09cb Automated merge with http://hg.scrapy.org/scrapy-0.10/ 2010-10-11 21:29:52 -02:00
Pablo Hoffman f5b188b179 Make RetryMiddleware obey Request.meta 'dont_retry' key when processing exceptions. Closes #259 2010-10-11 21:28:42 -02:00
Pablo Hoffman d5c8caf07b Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-10-10 20:31:38 -02:00
Pablo Hoffman b4fbc6c5fa Updated Scrapy Tutorial to reference feed exports, instead a custom written pipeline, and extended item pipeline documentation to include a JSON writer. 2010-10-10 20:31:05 -02:00
Pablo Hoffman 5267772d36 Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-10-09 20:43:47 -02:00
Pablo Hoffman 0b91c04007 Fixed issue with non-standard line ending in HTTP headers. Closes #258 2010-10-09 20:43:05 -02:00
Martin Olveyra a99bb0de9b Use binary flag in read/write operations, to fix tests in windows 2010-10-08 11:28:44 -02:00
Pablo Hoffman aa4142e4ba Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-10-07 18:23:48 -02:00
Pablo Hoffman f4accb6c7f Updated dmoz xpaths of Scrapy tutorial 2010-10-07 18:22:01 -02:00
Martin Olveyra a67a981cca Simplification of html parsing algorithm, fixed some tests (with new algorithm, comments inside bigger text region are generated separated from the text). Added test for a case not correctly handled by previous algorithm. Fixed test checking 2010-10-07 14:55:29 -02:00
Martin Olveyra fafaee51d5 htmlpage tests reorganization and fixes: improved how differences between expected and result are shown, and check also correct parsing of tag_type 2010-10-07 14:55:00 -02:00
Pablo Hoffman 571aeb559b Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-10-05 12:44:41 -02:00
Pablo Hoffman 4bbcbd7b77 Don't fail if twisted is not available on scrapy/__init__.py, to avoid making setup.py depend on Twisted. Closes #256 2010-10-05 12:43:34 -02:00
Pablo Hoffman 2d40705ea0 CloseSpider extension: Added support for closing spider after N errors have been raised. Closes #254 2010-09-30 20:17:44 -03:00
Pablo Hoffman b6e7a38a3a Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-09-29 13:37:33 -03:00
Pablo Hoffman 61ab9b86b7 Bumped version to 0.10.4 2010-09-29 13:36:36 -03:00
Pablo Hoffman ad0f180dd9 Added tag 0.10.3 for changeset 803efdb19e0b 2010-09-29 13:34:56 -03:00
Pablo Hoffman d15a97ff61 Updated Scrapy version in debian/changelog 2010-09-28 16:45:05 -03:00
Pablo Hoffman 7826869cb2 Added missing colon 2010-09-28 16:44:53 -03:00
Martin Santos 0bf9e4627c added support to CloseSpider extension, for close the spider after N pages have been crawled. Using the CLOSESPIDER_PAGECOUNT setting. closes #253 2010-09-28 16:29:37 -03:00
Pablo Hoffman 0976e0788e Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-09-27 12:27:58 -03:00
Pablo Hoffman 49ffe528a3 Fixed listen_tcp function when receiving None or 0 in portrange argument. Closes #252 2010-09-27 12:27:32 -03:00
Pablo Hoffman 50e57b08b0 Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-09-27 08:20:05 -03:00
Pablo Hoffman 9206806770 setup.py: added support for generating version from hg revision 2010-09-27 08:19:32 -03:00
Pablo Hoffman 51325fc93e Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-09-27 07:57:23 -03:00
Pablo Hoffman 52d198afc9 Removed forked cookielib tests, because Python cookielib has been suffering several changes and maintaining a fork of the tests has become a pain. Instead, we've added specific tests for the urllib2 request/response wrappers 2010-09-27 07:55:27 -03:00
Pablo Hoffman 3db8d367c4 Another fix to tests, broken by recent Python changes 2010-09-26 21:09:14 -03:00
Pablo Hoffman e55ae718e3 Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-09-26 17:00:10 -03:00
Pablo Hoffman 5d4e0655d0 Fixed test_utils_url, broken on Windows after recent Python urllib change. Closes #251 2010-09-26 16:57:42 -03:00
Pablo Hoffman 279dcc245f Fixed role name in Sphinx doc 2010-09-26 01:01:06 -03:00
Pablo Hoffman fd84d8d5c1 Automated merge with http://hg.scrapy.org/scrapy-0.10 2010-09-25 21:20:15 -03:00
Pablo Hoffman 1ef2cd400c Fixed issue with unicode keyword arguments bug in earlier Python versions (see http://bugs.python.org/issue2646). Closes #250 2010-09-25 21:17:36 -03:00
Pablo Hoffman 622834bc08 Removed unused imports, and use crawler.settings instead of scrapy.conf.settings in Scrapy Shell 2010-09-23 14:01:22 -03:00
Pablo Hoffman b78284b680 Fixed spider variable not properly populated in the Scrapy shell 2010-09-23 13:59:41 -03:00
Pablo Hoffman 318f7f4c58 Added support for passing code to evaluate in Scrapy shell command (closes #249) and simplified handling of shell errors 2010-09-23 13:49:29 -03:00
Pablo Hoffman a5ee05e814 Added support for setting exit code in Scrapy commands. Closes #248 2010-09-23 13:43:21 -03:00
Pablo Hoffman 79c0e34968 Simplified CrawlerSettings.__str__() 2010-09-23 12:50:46 -03:00
Pablo Hoffman 754d0f53f9 Fixed unbounded spider error in shell, and enclosed fetch() method in a try/except block for logging errors more reliably 2010-09-23 12:33:24 -03:00
Pablo Hoffman 37c25fe9a8 Fixed CrawlerSettings.__str__() method when settings_module is None 2010-09-23 12:32:49 -03:00
Pablo Hoffman f29b346f79 Fixed access to settings module name, broken after recent changes to Settings classes 2010-09-22 22:21:29 -03:00
Pablo Hoffman 9599bde3e9 Removed RequestLimitMiddleware 2010-09-22 16:09:13 -03:00