RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
4,408 Commits 49 Branches 132 Tags 138 MiB
Commit Graph

4408 Commits

Author SHA1 Message Date
tpeng a69f042d10 add 2 more test cases and minor doc fixes 2014-11-19 15:31:07 +01:00
tpeng fa84730e70 avoid download large response 
introduce DOWNLOAD_MAXSIZE and DOWNLOAD_WARNSIZE in settings and
download_maxsize/download_warnsize in spider/request meta, so
downloader stop downloading as soon as the received data exceed the
limit. also check the twsisted response's length in advance to stop
downloading as early as possible.
 2014-11-12 12:28:02 +01:00
Pablo Hoffman ed84231b60 Merge pull request #944 from JeffPaine/patch-1 
Update docs copyright year range
 2014-11-10 14:03:00 -02:00
Jeff Paine b422312a38 Update docs copyright year range 2014-11-09 21:08:27 -05:00
Lazar-T 13f83f0da0 typo 2014-11-10 06:28:41 +05:00
HalfCrazy b21a28cc9a Afterwords->Afterwards 2014-11-10 06:28:09 +05:00
Daniel Graña 2c67bd6c57 pywin32 is required by Twisted. closes #937 
see:
* http://twistedmatrix.com/trac/ticket/6032
* https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2028
 2014-11-05 23:05:54 -02:00
Daniel Graña 6cb8995731 Update install.rst 
fixes #937
 2014-11-05 22:50:52 -02:00
Pablo Hoffman efe589c643 Merge pull request #882 from ahlen/feature/csvfeed-quotechar 
[MRG+1] Allow to specify the quotechar in CSVFeedSpider
 2014-11-04 11:32:59 -02:00
Lazar-T 38dcf50cd6 comma instead of fullstop 2014-10-25 09:19:50 +06:00
Pablo Hoffman 675fd5ba04 Merge pull request #898 from scrapy/download-timeout 
[MRG] DOC document download_timeout
 2014-10-24 16:52:42 -02:00
Pablo Hoffman 5de6a11fda Merge pull request #925 from Digenis/master 
a leftover for.15 compatibility
 2014-10-22 18:23:19 -02:00
Nikolaos-Digenis Karagiannis 2227805619 Compatibility with .15 leftover 2014-10-22 21:48:08 +03:00
Pablo Hoffman 0dce283459 Merge pull request #893 from kmike/less-ads 
[MRG] DOC simplify extension docs
 2014-10-21 17:13:59 -02:00
Daniel Graña 44cbbecb44 Merge pull request #914 from brunsgaard/master 
Deleted bin folder from root, fixes #913
 2014-10-09 14:58:36 -02:00
Pablo Hoffman aa61f615d8 Merge pull request #795 from chekunkov/spider_error_processing_referer 
Add referer to "Spider error processing" log message
 2014-10-07 20:57:48 -02:00
Jonas Brunsgaard db2474f7e7 Deleted bin folder from root, fixes #913 2014-10-07 13:54:04 +02:00
Mikhail Korobov 7d68b084a4 DOC document download_timeout Request.meta key and download_timeout spider attribute. 2014-10-07 04:23:11 +06:00
Pablo Hoffman 9af61d5df6 Merge pull request #895 from scrapy/we-are-past-0.15 
[MRG] drop support for CONCURRENT_REQUESTS_PER_SPIDER
 2014-10-03 17:14:00 -03:00
Pablo Hoffman 5f1bbe2dd3 Merge pull request #911 from nyov/nyov/dev 
Drop old engine code
 2014-10-03 17:13:17 -03:00
nyov 7db6bbce27 Drop old engine code 
* remove Downloader import unused since 1fba64
  * remove CONCURRENT_SPIDERS deprecation warning from a1dbc6 (2011)
 2014-10-03 19:07:51 +00:00
Mikhail Korobov 14957ed71e Merge pull request #909 from VKen/master 
updated deprecated cgi.parse_qsl to use six's parse_qsl
 2014-10-03 16:30:39 +06:00
VKen 33a7c1d438 updated deprecated cgi.parse_qsl to use six's parse_qsl 2014-10-03 04:16:21 +08:00
Mikhail Korobov ea3b372b4f DOC typo fix in leaks.rst 2014-10-02 15:20:13 +06:00
Pablo Hoffman 993b543e1b mark SEP-019 as Final 2014-10-02 01:17:31 -03:00
Pablo Hoffman e7843d35de Merge pull request #894 from kmike/leaks-docs 
Leaks docs
 2014-10-02 01:14:54 -03:00
Pablo Hoffman 5835224eee Merge pull request #896 from scrapy/robotstxt-once 
[MRG] process robots.txt once
 2014-10-02 00:58:55 -03:00
Pablo Hoffman 9e2c60430d Merge pull request #902 from scrapy/load_object_full_traceback 
[MRG] scrapy.utils.misc.load_object should print full traceback
 2014-10-02 00:33:27 -03:00
Pablo Hoffman 0c23b1342b Merge pull request #904 from scrapy/from_crawler_docs 
[MRG] DOC document from_crawler method for item pipelines
 2014-10-02 00:09:07 -03:00
Mikhail Korobov 6fcf9dce50 DOC document from_crawler method for item pipelines; add an example. 2014-09-25 03:13:51 +06:00
Mikhail Korobov 5086262913 don't hide original exception in scrapy.utils.misc.load_object 2014-09-24 13:27:14 +06:00
Mikhail Korobov 36eec8f413 dont_obey_robotstxt meta key; don't process requests to /robots.txt 2014-09-23 00:10:43 +06:00
Mikhail Korobov fe6f3efe95 RobotsTxtMiddleware: remove unused attribute 2014-09-22 22:56:54 +06:00
Mikhail Korobov d11c8595e6 drop support for CONCURRENT_REQUESTS_PER_SPIDER 2014-09-22 04:29:22 +06:00
Mikhail Korobov bdbca1e2d7 DOC request queue memory usage 2014-09-21 07:30:44 +06:00
Mikhail Korobov bc0f481a73 DOC bring back notes about multiple spiders per process because it is now documented how to do that 2014-09-21 07:12:01 +06:00
Mikhail Korobov a122fdbfea Update leaks.rst: there is now only a single spider in a process. 2014-09-21 06:54:00 +06:00
Mikhail Korobov 7be3479c20 CookieJar cleanup 2014-09-21 06:37:32 +06:00
Mikhail Korobov 49645d4bf9 TST small cleanup of a cookie test 2014-09-21 05:31:34 +06:00
Mikhail Korobov c543fe6e4c Merge pull request #878 from andrewshir/master 
Fix bug for ".local" host name
 2014-09-21 05:24:54 +06:00
Mikhail Korobov e435b3e3a3 DOC simplify extension docs 2014-09-21 00:19:24 +06:00
John-Scott Atlakson a312ebfb43 Update request-response.rst 
Fixed minor typo
 2014-09-14 22:06:31 +06:00
andrewshir e583c030db Test for local domains (without dots) added 2014-09-14 14:24:16 +06:00
Mikael Åhlén 22da1783bd added a test-case for wrong quotechar 2014-09-13 03:47:40 +02:00
Mikael Åhlén 47b6dff9f1 Allow to specify the quotechar in CSVFeedSpider 2014-09-13 02:14:57 +02:00
Daniel Graña 5bcabfe9c9 SPIDER_MODULES can be set as a csv string 2014-09-10 23:25:57 -03:00
Daniel Graña c05e99a4f4 oops, restore Pillow from precise test requirements 2014-09-10 12:21:08 -03:00
Daniel Graña a823207f18 Stop logobserver only when set 2014-09-10 12:09:07 -03:00
Daniel Graña ec93c0fdcc Add the tests changes for previous commit 2014-09-10 12:05:18 -03:00
Daniel Graña ce180227fa Twisted 11.1.0 (precise) can not deal with generators in DeferredList 
Also create a list of the crawlers before iterating them because crawlers are removed from the set once stopped
 2014-09-10 12:04:14 -03:00