RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
2,882 Commits 49 Branches 132 Tags 144 MiB
Commit Graph

2882 Commits

Author SHA1 Message Date
Daniel Graña 7cba9bc274 add travis-ci configuration for multiple build enviroments 2012-05-17 09:38:47 -03:00
Daniel Graña 0d4dccd0fc bump version to 0.14.4 2012-05-15 15:28:45 -03:00
Pablo Hoffman b7e46dfdf4 added precise to supported ubuntu distros 2012-05-12 19:54:45 -03:00
Pablo Hoffman 340fbdbda0 fixed bug in json-rpc webservice reported in https://groups.google.com/d/topic/scrapy-users/qgVBmFybNAQ/discussion. also removed no longer supported 'run' command from extras/scrapy-ws.py 2012-05-03 12:08:03 -03:00
Daniel Graña 0cb68afe7e meta tag attributes for content-type http equiv can be in any order. #123 2012-04-27 01:04:00 -03:00
Pablo Hoffman 4d17048ac2 replace "import Image" by more standard "from PIL import Image". closes #88 2012-04-20 19:04:44 -03:00
Daniel Graña b7b2e7fa98 return trial status as bin/runtests.sh exit value. #118 2012-04-20 16:41:31 -03:00
Daniel Graña 02df475711 bumped version to 0.14.3 2012-04-19 23:06:55 -03:00
Daniel Graña fd85f9cc8e forgot to include pydispatch license. #118 2012-04-19 22:56:59 -03:00
Daniel Graña c89779363a include egg files used by testsuite in source distribution. #118 2012-04-19 22:40:02 -03:00
Pablo Hoffman 2548dccbcb update docstring in project template to avoid confusion with genspider command, which may be considered as an advanced feature. refs #107 2012-04-19 02:15:30 -03:00
Pablo Hoffman 668e352440 added note to docs/topics/firebug.rst about google directory being shut down 2012-04-19 01:33:17 -03:00
Pablo Hoffman 835d082294 Merge branch '0.14' of github.com:scrapy/scrapy into 0.14 2012-04-10 15:52:43 -03:00
olveyra 8e9f607747 dont discard slot when empty, just save in another dict in order to recycle if needed again. 
This fix avoids to continuosly create new slot under certain cases, bug that prevents download_delay and max_concurrent_requests to work properly.

The problem arises when the slot for a given domain becomes empty, but further requests for that domain werent still created by the spider. This is typical when spider creates requests one by one, or it makes requests to multiple domains and one or more of them are created in a rate enough slow that makes slot to be empty each time the response is fetched.

The effect is that a new slot is created for each request under such conditions, and so the download_delay and max_concurrent_requests are not taking effect (because in order to apply, depends on an already existing slot for that domain).
 2012-04-09 01:10:38 -03:00
Daniel Graña b830e95e15 do not fail handling unicode xpaths in libxml2 backed selectors 2012-04-04 17:21:18 -03:00
Jason Yeo bf3c9eece2 fixed minor mistake in Request objects documentation 2012-03-20 23:50:21 -03:00
Pablo Hoffman ba14f381f9 fixed minor defect in link extractors documentation 2012-03-20 22:56:56 -03:00
Pablo Hoffman 0665175d3b removed some obsolete remaining code related to sqlite support in scrapy 2012-03-16 12:04:41 -03:00
Pablo Hoffman 68c1c98d28 bumped version to 0.14.2 2012-03-11 12:58:58 -03:00
Daniel Graña 6a5bef291d move buffer pointing to start of file before computing checksum. refs #92 2012-02-23 19:39:08 -02:00
Daniel Graña 9817df1078 Compute image checksum before persisting images. closes #92 
Avoids threading issue accesing buffer
 2012-02-23 19:39:05 -02:00
Daniel Graña 673a120e99 remove leaking references in cached failures 2012-02-23 19:38:13 -02:00
Pablo Hoffman 11133e9fb9 fixed bug in MemoryUsage extension: get_engine_status() takes exactly 1 argument (0 given) 2012-02-19 05:59:48 -02:00
Pablo Hoffman 1627320264 Merge branch '0.14' of github.com:scrapy/scrapy into 0.14 2012-02-11 20:29:51 -02:00
Pablo Hoffman 14231409d7 fixed struct.error on http compression middleware. closes #87 2012-02-11 20:28:20 -02:00
Daniel Graña 0de3fb414f ajax crawling wasn't expanding for unicode urls 2012-02-07 14:50:32 -02:00
Daniel Graña 454a21db90 Catch start_requests iterator errors. refs #83 2012-02-07 14:50:09 -02:00
Сергей Прохоров 2fbd662ab7 Speed-up libxml2 XPathSelector 2012-01-16 01:42:56 -02:00
Pablo Hoffman 0a070f57a7 updated versioning doc according to recent changes 2012-01-05 11:50:39 -02:00
Pablo Hoffman 2b4e4c37a0 scrapyd: fixed documentation link 2012-01-03 22:22:49 -02:00
Pablo Hoffman caffe0e800 extras/makedeb.py: no longer obtaining version from git 2012-01-02 13:28:24 -02:00
Pablo Hoffman 6cb9e1c6a0 bumped version to 0.14.1 2012-01-02 13:15:27 -02:00
Pablo Hoffman 4b86bd657e fixed reference to tutorial directory 2011-12-23 18:57:23 -02:00
Pablo Hoffman 1aeccdd6c5 doc: removed duplicated callback argument from Request.replace() 2011-12-23 15:56:01 -02:00
Pablo Hoffman 8bf19e6b41 fixed formatting of scrapyd doc 2011-12-20 03:18:58 -02:00
Daniel Graña 14a8e6e64c Dump stacks for all running threads and fix engine status dumped by StackTraceDump extension 2011-12-15 18:08:59 -02:00
Pablo Hoffman 5223575352 added comment about why we disable ssl on boto images upload 2011-12-15 17:06:36 -02:00
Daniel Graña 63d583d9be SSL handshaking hangs when doing too many parallel connections to S3 2011-12-14 13:06:12 -02:00
Daniel Graña bcb31988f2 change tutorial to follow changes on dmoz site 2011-12-14 13:03:31 -02:00
Rolando Espinoza La fuente 98f3f87530 Avoid _disconnectedDeferred AttributeError exception in Twisted>=11.1.0 2011-12-12 18:11:43 +00:00
Martin Olveyra 175a4b5957 allow spider to set autothrottle max concurrency 2011-12-02 15:50:53 -02:00
Pablo Hoffman 0f649f0e30 bumped version to 0.14 2011-11-17 14:43:40 -02:00
Pablo Hoffman 6d13de4366 fixed "No free spider slots" bug when calling fetch() from scrapy shell 2011-11-14 20:03:43 -02:00
Pablo Hoffman d37a788d22 improve handling of KeyError exception when creating spiders in spider manager. closes issue 49 2011-11-14 17:00:25 -02:00
Pablo Hoffman 36df87b4de ignore meta-refresh redirects embedded in <script> tags. related to issue 18 2011-11-14 16:54:13 -02:00
Pablo Hoffman ec1ef0235f ignore meta-refresh redirect when embedded inside <noscript> tag. closes issue 18 2011-11-14 16:25:22 -02:00
Pablo Hoffman 6cc40dc062 fixed bug in MEMUSAGE_NOTIFY_MAIL setting 2011-11-08 11:51:26 -02:00
Pablo Hoffman 37ad4f8791 added support for ajax crawleable urls 2011-10-28 16:33:12 -02:00
Pablo Hoffman 992af8d38f ubuntu repos: added support for oneiric release 2011-10-25 14:26:38 -02:00
Pablo Hoffman f4821a123d Do not raise PartialDownloadError if Content-Length doesn't match the body size. This fixes the error reported in: https://groups.google.com/d/topic/scrapy-users/FQ25O3KPQuU/discussion 2011-10-25 13:04:58 -02:00