RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
2,962 Commits 49 Branches 132 Tags 143 MiB
Commit Graph

2962 Commits

Author SHA1 Message Date
Daniel Graña 18a35a9fd6 Merge branch 'lxml-selectors' 2012-04-13 09:32:51 -03:00
Daniel Graña 3dbe211d29 lxml boolean results fix. oops #116 2012-04-13 09:29:56 -03:00
Daniel Graña a338c29287 Merge pull request #116 from dangra/lxml-selectors 
more fixes to lxml selector incompatibilities
 2012-04-13 04:38:45 -07:00
Daniel Graña b9efa5ee73 more fixes to lxml selector incompatibilities 
* Do not fail parsing empty bodies
* Do not fail parsing bodies with null bytes
* Recode to utf8 using response.body_as_unicode() to avoid decoding bugs
* Return empty results with unevaluable nodes like text or attribute nodes
* Return u'1' and u'0' for boolean xpaths
 2012-04-13 00:58:31 -03:00
Daniel Graña d8ebf16fe5 Merge pull request #114 from stav/master 
Scrapy DOC changes
 2012-04-11 12:08:45 -07:00
Pablo Hoffman 7cca916ed5 added release notes to official documentation, including all release notes since Scrapy 0.7 2012-04-11 15:53:23 -03:00
stav f1802289cd small doc typo change to get the fork rolling 2012-04-11 12:05:39 -05:00
Daniel Graña 02833e3265 fix typo in module description. closes #112 2012-04-11 10:44:31 -03:00
Pablo Hoffman 4f28ffcb2c removed no longer needed dependency on simplejson 2012-04-10 16:01:36 -03:00
Pablo Hoffman 6e8edbd72e switched default selectors backend to lxml 2012-04-10 15:52:14 -03:00
Daniel Graña af0e1c40f5 Avoid logging useless error messages about ignored requests in robots.txt 2012-04-10 13:37:32 -03:00
Daniel Graña 97e4003a56 do not fail handling unicode xpaths in libxml2 backed selectors 2012-04-04 17:18:31 -03:00
Pablo Hoffman ab4dd928ee Merge pull request #108 from kalessin/throttleslot 
Fix autothrottle in order to modify also inactive downloader slots, so c...
 2012-04-03 17:02:31 -07:00
olveyra e6d7afa13b Fix autothrottle in order to modify also inactive downloader slots, so cases fixed by inactive slots patch will work ok also when using autothrottle 2012-04-03 23:14:00 +00:00
Pablo Hoffman c27f7eb7e9 Merge pull request #106 from kalessin/downloader2 
dont discard slot when empty, just save in another dict in order to recycle if needed again
 2012-04-02 14:39:38 -07:00
olveyra b39cb22d83 dont discard slot when empty, just save in another dict in order to recycle if needed again. 
This fix avoids to continuosly create new slot under certain cases, bug that prevents download_delay and max_concurrent_requests to work properly.

The problem arises when the slot for a given domain becomes empty, but further requests for that domain werent still created by the spider. This is typical when spider creates requests one by one, or it makes requests to multiple domains and one or more of them are created in a rate enough slow that makes slot to be empty each time the response is fetched.

The effect is that a new slot is created for each request under such conditions, and so the download_delay and max_concurrent_requests are not taking effect (because in order to apply, depends on an already existing slot for that domain).
 2012-04-02 20:34:57 +00:00
Pablo Hoffman e9184def35 make selector re() method use re.UNICODE flag to compile regexes 2012-04-01 00:41:03 -03:00
Pablo Hoffman 27018fced7 changed default user agent to Scrapy/0.15 (+http://scrapy.org) and removed no longer needed BOT_VERSION setting 2012-03-23 13:45:21 -03:00
Pablo Hoffman 731c569b5c fixed test-scrapyd.sh script after changed on insophia website 2012-03-22 16:38:28 -03:00
Pablo Hoffman 8933e2f2be added REFERER_ENABLED setting, to control referer middleware 2012-03-22 16:35:14 -03:00
Pablo Hoffman eed34e88cd Merge pull request #103 from jsyeo/patch-1 
fixed minor mistake in Request objects documentation
 2012-03-20 19:49:31 -07:00
Jason Yeo da826aa13d fixed minor mistake in Request objects documentation 2012-03-21 10:25:41 +08:00
Pablo Hoffman 175c70ad44 fixed minor defect in link extractors documentation 2012-03-20 22:56:45 -03:00
Pablo Hoffman 056a7c53d0 added artwork files properly now 2012-03-20 10:46:45 -03:00
Pablo Hoffman aef70e8394 removed wrongly added artwork files 2012-03-20 10:45:48 -03:00
Pablo Hoffman bcd8520f8d added sep directory with Scrapy Enhancement Proposal imported from old Trac site 2012-03-20 10:15:00 -03:00
Pablo Hoffman c0141d154e added artwork directory (data taken from old Trac) 2012-03-20 10:14:11 -03:00
Pablo Hoffman 35fb01156e removed some obsolete remaining code related to sqlite support in scrapy 2012-03-16 11:55:55 -03:00
Pablo Hoffman 838e1dcce9 updated FormRequest tests to use HtmlResponse instead of Response, as it makes more sense 2012-03-15 11:47:02 -03:00
Pablo Hoffman b6ae266546 Removed (very old and possibly broken) backwards compatibility support for Twisted 2.5 2012-03-15 00:28:24 -03:00
Pablo Hoffman 9fddc73ed8 removed backwards compatibility code for old scrapy versions 2012-03-06 05:42:09 -02:00
Pablo Hoffman 9a508d4638 Removed deprecated setting: CLOSESPIDER_ITEMPASSED 2012-03-06 05:26:57 -02:00
Pablo Hoffman 8b83177655 Added CLOSESPIDER_ERRORCOUNT to scrapy/default_settings.py 2012-03-06 05:26:57 -02:00
Pablo Hoffman 9006227358 bumped required python-w3lib version in debian/control 2012-03-05 20:25:38 -02:00
Daniel Graña 2909a60e95 test that default start_request return value type is a generator. refs #98 2012-03-05 17:53:20 -02:00
Pablo Hoffman 45685ea6cd Restored scrapy.utils.py26 module for backwards compatibility, with a deprecation message. This is needed because the module was used a lot by users and the change causes too much trouble 2012-03-05 17:15:49 -02:00
Daniel Graña cc6e297062 Merge pull request #98 from kalessin/start_requests 
This will break any spider that extends `start_requests` and expect a `list` as return value.

In the other side:

* [Docs](http://doc.scrapy.org/en/latest/topics/spiders.html#scrapy.spider.BaseSpider.start_requests) says that return value is **iterable** not list: 
* Scrapy core already support consuming start_requests generator on demand so we can avoid problems like #47
* it allows extensions to change starting requests on `spider_opened` signal
 2012-03-05 08:51:22 -08:00
Martin Olveyra f6179a927e replace list by generator also in start_requests method of Sitemap 
spider
 2012-03-05 14:25:12 -02:00
Martin Olveyra cc7fc33833 change start_request to return a generator instead of a list, in order 
to allow to modify start_urls triggered by spider_opened signal
 2012-03-05 12:49:17 -02:00
Pablo Hoffman e521da2e2f Dropped support for Python 2.5. See: http://blog.scrapy.org/scrapy-dropping-support-for-python-25 2012-03-01 08:18:12 -02:00
Pablo Hoffman 8eb0b11f8a removed unused import 2012-02-29 17:40:30 -02:00
Pablo Hoffman 5c329b6514 Merge pull request #97 from scrapy/w3lib_encoding 
Ported scrapy to use w3lib.encoding
 2012-02-29 01:45:59 -08:00
Pablo Hoffman de3a3b68dc bumped required w3lib version to 1.1, after refactoring encoding detection to use the new w3lib.encoding module 2012-02-29 07:44:22 -02:00
Pablo Hoffman 2b16ebdc11 added minor clarification on cookiejar request meta key usage 2012-02-29 07:19:01 -02:00
Pablo Hoffman 61df6b4691 Merge pull request #51 from lostsnow/master 
scrapyd: support bind to a specific ip address
 2012-02-28 23:49:56 -08:00
lostsnow 5afe4f50c1 scrapyd: support bind to a specific ip address 2012-02-29 13:47:40 +08:00
Daniel Graña 798169805a Adapt response encoding detection to pass test cases 2012-02-28 14:32:55 -02:00
Pablo Hoffman 81abb45000 fixed bug in new cookiejar documentation 2012-02-28 11:08:25 -02:00
Pablo Hoffman 26c8004125 added documentation for the new cookiejar Request.meta key 2012-02-27 19:58:58 -02:00
Pablo Hoffman 44d6da82fd Merge pull request #96 from kalessin/cookiesmultijar 
allow to work with multiple cookie jars on the same spider
 2012-02-27 13:48:43 -08:00