RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
3,016 Commits 49 Branches 132 Tags 129 MiB
Commit Graph

3016 Commits

Author SHA1 Message Date
Rolando Espinoza La fuente 93eb5b32dd tests: fixed httpcache testcase. 2011-12-30 23:11:16 -04:00
Pablo Hoffman a36c8691d7 Merge pull request #74 from darkrho/httpcache-stats 
httpcache: keep stats of cache hit/miss/store and don't store already cached response
 2011-12-30 18:15:43 -08:00
Rolando Espinoza La fuente 503fdf39fe httpcache: don't store already cached response. 2011-12-30 14:21:07 -04:00
Rolando Espinoza La fuente f2966eebc7 httpcache: keep stats of cache hit/miss/store. 2011-12-30 14:10:18 -04:00
Pablo Hoffman 9064188035 removed unused import 2011-12-28 15:21:10 -02:00
Pablo Hoffman 150f82e600 some some changes to scrapyd listjobs.json api: 
* the api is now a GET instead of POST (for consistency)
* the api also returns pending and finished jobs, in addition to running
  ones
* only the last 100 finished jobs are kept (can be changed through the
  finished_to_keep setting)
 2011-12-28 15:17:52 -02:00
Pablo Hoffman 1dfbe5d7a8 scrapyd.webservice: relocate ListJobs resource for better consistency 2011-12-28 14:36:24 -02:00
Pablo Hoffman f214c94912 CrawlSpider: don't follow links from non-HTML responses 2011-12-27 21:22:38 -02:00
Pablo Hoffman bda9c97c78 Merge pull request #48 from simonratner/delete-logs-by-mtime 
Delete old logs based on file mtime.
 2011-12-23 13:17:00 -08:00
Pablo Hoffman 0be421fbf0 fixed reference to tutorial directory 2011-12-23 18:57:11 -02:00
Pablo Hoffman 41fd3c4f6c doc: removed duplicated callback argument from Request.replace() 2011-12-23 15:55:46 -02:00
Pablo Hoffman 0eeff76227 fixed formatting of scrapyd doc 2011-12-20 03:18:37 -02:00
Daniel Graña 64ba6e7982 Dump stacks for all running threads and fix engine status dumped by StackTraceDump extension 2011-12-15 17:05:45 -02:00
Pablo Hoffman 023232f7d4 added comment about why we disable ssl on boto images upload 2011-12-15 14:23:45 -02:00
Daniel Graña aea060e144 Merge branch '0.14' 2011-12-14 13:09:00 -02:00
Daniel Graña 63d583d9be SSL handshaking hangs when doing too many parallel connections to S3 2011-12-14 13:06:12 -02:00
Daniel Graña bcb31988f2 change tutorial to follow changes on dmoz site 2011-12-14 13:03:31 -02:00
Rolando Espinoza La fuente 98f3f87530 Avoid _disconnectedDeferred AttributeError exception in Twisted>=11.1.0 2011-12-12 18:11:43 +00:00
Pablo Hoffman 9b84914736 Merge pull request #62 from darkrho/issue-58 
Avoid _disconnectedDeferred AttributeError exception in Twisted>=11.1.0
 2011-12-12 10:10:08 -08:00
Rolando Espinoza La fuente 9c04945785 Avoid _disconnectedDeferred AttributeError exception in Twisted>=11.1.0 2011-12-04 23:42:41 -04:00
Martin Olveyra 175a4b5957 allow spider to set autothrottle max concurrency 2011-12-02 15:50:53 -02:00
Pablo Hoffman 4fe42dc6fe Merge pull request #61 from kalessin/master 
allow spider to set autothrottle max concurrency
 2011-12-01 12:47:14 -08:00
Martin Olveyra 7b0184eb59 allow spider to set autothrottle max concurrency 2011-12-01 18:44:26 -02:00
Pablo Hoffman e29f9e5b24 bumped version to 0.15 2011-11-17 14:44:34 -02:00
Pablo Hoffman 0f649f0e30 bumped version to 0.14 2011-11-17 14:43:40 -02:00
Pablo Hoffman 6d13de4366 fixed "No free spider slots" bug when calling fetch() from scrapy shell 2011-11-14 20:03:43 -02:00
Pablo Hoffman d37a788d22 improve handling of KeyError exception when creating spiders in spider manager. closes issue 49 2011-11-14 17:00:25 -02:00
Pablo Hoffman 36df87b4de ignore meta-refresh redirects embedded in <script> tags. related to issue 18 2011-11-14 16:54:13 -02:00
Pablo Hoffman ec1ef0235f ignore meta-refresh redirect when embedded inside <noscript> tag. closes issue 18 2011-11-14 16:25:22 -02:00
Simon Ratner 7232c31f78 Delete old logs based on file mtime. 2011-11-11 11:53:00 -08:00
Pablo Hoffman 6cc40dc062 fixed bug in MEMUSAGE_NOTIFY_MAIL setting 2011-11-08 11:51:26 -02:00
Pablo Hoffman 37ad4f8791 added support for ajax crawleable urls 2011-10-28 16:33:12 -02:00
Pablo Hoffman 992af8d38f ubuntu repos: added support for oneiric release 2011-10-25 14:26:38 -02:00
Pablo Hoffman f4821a123d Do not raise PartialDownloadError if Content-Length doesn't match the body size. This fixes the error reported in: https://groups.google.com/d/topic/scrapy-users/FQ25O3KPQuU/discussion 2011-10-25 13:04:58 -02:00
Pablo Hoffman c085f81641 removed deprecation warning for spider.download_timeout attribute 2011-10-25 04:26:37 -02:00
Pablo Hoffman c38c49d56a fixed PickeItemExporter bug, added unittest, and added pickle to suported feed exports formats 2011-10-25 02:36:51 -02:00
Pablo Hoffman 8bdf288428 made scrapyd doc more version agnostic 2011-10-23 05:29:54 -02:00
Pablo Hoffman 64b8e2648e added support for using '-' in scrapy crawl -o, to dump items to standard output 2011-10-23 03:06:59 -02:00
Pablo Hoffman 028bf3386d feed exports: removed dependency on file.tell() method, so that stdout output works 2011-10-23 03:05:06 -02:00
Pablo Hoffman 10ced29e18 changed feed exports storage api so that file/stdio outputs directly without using a temporary file 2011-10-23 02:49:17 -02:00
Pablo Hoffman ade5efdc61 added -o option to scrapy crawl, a convenient shortcut for using feed exports 2011-10-22 20:53:49 -02:00
Pablo Hoffman 13cd9a1b0f remove deprecation warning for spider.user_agent attribute 2011-10-22 19:28:12 -02:00
Pablo Hoffman 43b79afc9c remove usage of assertLess() which is only available on python 2.7+ 2011-09-26 12:21:52 -03:00
Pablo Hoffman 431441cb52 updated documentation to remove references to old issue tracker and mercurial repos 2011-09-25 13:06:24 -03:00
Pablo Hoffman ce03ccd4ec updated documentation about DEPTH_PRIORITY and DFO/BFO crawls 2011-09-23 13:22:25 -03:00
Pablo Hoffman f850a44784 Some changes to persistent scheduler after some initial usage feedback: 
* added LIFO queues, in addition to the original FIFO queues
* use LIFO queues (instead of FIFO queues) by default, since they resemble DFO
  better which is a more convenient crawling order for most cases
* do not adjust the priority based on depth by default (DEPTH_PRIORITY = 0)

If someone does need to use strict BFO order, it can be by done by setting:

    DEPTH_PRIORITY = 1
    SCHEDULER_DISK_QUEUE = 'scrapy.squeue.PickleFifoDiskQueue'
    SCHEDULER_MEMORY_QUEUE = 'scrapy.squeue.FifoMemoryQueue'
 2011-09-23 13:03:07 -03:00
Pablo Hoffman cfddc314ce make SpiderState extension always available, regardless of whether there is a job dir, and make sure it always set spider.state attribute, for consistency between in-memory and on-disk runs 2011-09-23 12:56:44 -03:00
Pablo Hoffman 2559f211a8 Merge pull request #42 from noplay/deploy-git-version 
scrapy deploy support git version
 2011-09-22 14:42:41 -07:00
Julien Duponchelle b7c436343a scrapy deploy support git version 2011-09-21 22:17:08 +02:00
Pablo Hoffman 0f0783e525 Merge pull request #39 from kalessin/master 
autothrottle fix
 2011-09-21 13:11:28 -07:00