RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
3,752 Commits 49 Branches 132 Tags 138 MiB
Commit Graph

3752 Commits

Author SHA1 Message Date
Pablo Hoffman 80bb9fb5d2 Merge pull request #497 from kmike/utils-unique 
[MRG] no need to use dict in scrapy.utils.python.unique
 2013-12-19 05:40:34 -08:00
Mikhail Korobov 84aa7599a4 no need to use dict in scrapy.utils.python.unique 2013-12-19 15:06:27 +06:00
Mikhail Korobov 93eee9d1c6 Merge pull request #479 from citizen-stig/unicode_spider_name 
fix logging error with unicode spider name
 2013-12-18 12:17:57 -08:00
Mikhail Korobov 51f340e1b9 add distutils folders to gitignore 2013-12-18 23:29:07 +06:00
Pablo Hoffman 462e40acd0 Merge pull request #489 from RasPat1/patch-2 
Note about selector class import
 2013-12-16 06:40:59 -08:00
RasPat1 ff21281b95 Note about selector class import 
This is the salient point of this code compared to the last example.  We have a selector now and this is how we use it.  Especially since the user has just come from the shell where the pre-instantiated selector is taken for granted.
 2013-12-15 13:46:42 -05:00
Daniel Graña 8a7c5b5d81 Add 0.20.2 release notes 
Conflicts:
	docs/news.rst
 2013-12-09 18:33:46 -02:00
Daniel Graña 11359188a5 Merge pull request #484 from nyov/nyov/crawl-tmpl-selector 
Update CrawlSpider Template with Selector changes
 2013-12-09 12:21:27 -08:00
nyov b6a200d02a Update CrawlSpider Template with Selector changes 2013-12-06 20:48:25 +00:00
Daniel Graña 72543c9ef0 Merge pull request #397 from duendex/duendex/proxyTunnel 
Adds the functionality to do HTTPS downloads behind proxies using an
 2013-12-03 14:48:42 -08:00
duendex 8ada8f5f36 Added a test case to ensure that passing the noconnect paramenter avoids trigerring the creation of a connect tunnel when downloading from a site with https scheme. 2013-12-03 12:55:44 -02:00
duendex 6427d60fd8 Fixed the location of the certificate required by libmproxy. 2013-12-03 11:45:02 -02:00
duendex 500490ee73 Corrected a test that used a dummy URL that unpurposedly had an https scheme and failed with PR 397. 2013-12-03 03:10:16 -02:00
duendex f8dea74948 Added a delay to wait for the proxy to start. 2013-12-03 03:09:06 -02:00
duendex 247b330f08 Corrected typo in tox.ini 2013-12-02 21:34:35 -02:00
duendex 02bab270e8 Added mitmproxy as a requirement. 2013-12-02 20:25:12 -02:00
duendex d69ba7c1ae Changed the proxy tests to use libmproxy instead of starting mitmdump as a separate process. 2013-12-02 20:21:43 -02:00
duendex 88bec496f2 The response matching re is now compiled once at module load time. 2013-12-02 20:19:42 -02:00
duendex 23c3288a6d Adds the option to omit the usage of a CONNECT tunnel by adding the noconnect 
parameter to the URL of the proxy.
 2013-12-02 20:19:42 -02:00
duendex 7f053cc1d2 Adds support for proxy authentication when openning a CONNECT tunnel. 2013-12-02 20:19:42 -02:00
duendex 628bfbcc3e Raises a custom TunnelError when the tunnel cannot be opened. Removed unnecesary comments. 2013-12-02 20:19:42 -02:00
duendex 58a98b0c04 Improved error handling. 2013-12-02 20:19:42 -02:00
duendex 36e4fc3785 Removed some trailing spaces that I left. 2013-12-02 20:19:42 -02:00
duendex ae28c7d698 Adds the functionality to do HTTPS downloads behind proxies using an 
HTTP CONNECT.
 2013-12-02 20:19:41 -02:00
Pablo Hoffman f2741c413e fix method name in tutorial. closes GH-480 2013-12-02 13:24:12 -02:00
Nikolay Golub a651a75248 fix logging error with unicode spider name 
Log message fails if spider name is in unicode, because "system" key in eventDict isn't encoded.
 2013-11-30 21:08:56 +04:00
Daniel Graña e34ffc0f42 Add 0.20.1 release notes 
Conflicts:
	docs/news.rst
 2013-11-28 16:25:57 -02:00
Daniel Graña cfe588103c include_package_data is required to build wheels from published sources 2013-11-28 16:25:57 -02:00
Pablo Hoffman 339861367e Merge pull request #425 from audiodude/master 
DownloaderMiddleware docs: Update process_request and minor cleanups.
 2013-11-25 10:33:35 -08:00
Daniel Graña aeeba2147b travis-ci updated to pypy 2.2 2013-11-25 13:57:40 -02:00
Daniel Graña 36c8da2ad6 Merge pull request #461 from redapple/selectorloader 
Add "unified" SelectorItemLoader (supports .add_css() and .add_xpath())
 2013-11-22 12:10:39 -08:00
Pablo Hoffman 545f2601b0 Merge pull request #469 from redapple/xgzip 
Remove "x-gzip" from Requests' "Accept-Encoding" header
 2013-11-21 10:55:07 -08:00
Paul Tremberth 0d99babe40 Remove "x-gzip" from Requests' "Accept-Encoding" header 2013-11-21 18:58:24 +01:00
Paul Tremberth 14f5817d6b Modify ItemLoader to support XPath and CSS selectors 
Deprecate XPathItemLoader (now an alias to the new ItemLoader)
 2013-11-21 18:05:24 +01:00
Pablo Hoffman f87be371a2 better names for HANDLE_* settings, and added doc 2013-11-21 14:33:17 -02:00
Daniel Graña ab01e9e9e4 Merge pull request #466 from kalessin/httperror 
allow to use settings for defining http error handling defaults
 2013-11-21 03:55:44 -08:00
Martin Olveyra 55bee912a2 allow to use settings for defining http error handling defaults 2013-11-20 20:12:49 -02:00
Mikhail Korobov 8416cc7515 Merge pull request #465 from bjlange/master 
Add note to item-pipeline documentation explaining order
 2013-11-20 09:40:19 -08:00
Brian Lange e4c1d8d37d Elaborate on use of order numbers 2013-11-19 17:51:50 -06:00
Daniel Graña 2564c21d4c add a tox env for Python 3.3 2013-11-19 20:15:15 -02:00
Daniel Graña 526a944eda lxml is required, no need to skip tests. 2013-11-19 20:14:48 -02:00
Daniel Graña 3f156ad845 Do not call body_as_unicode on non text responses. closes #462 2013-11-19 20:13:34 -02:00
Brian Lange b878f60b5a Add note to item-pipeline documentation explaining order in the ITEM_PIPELINES setting. 2013-11-19 16:12:54 -06:00
Daniel Graña ec7833a910 Deprecate body_or_str helper function only used by xml iterators 2013-11-19 19:21:54 -02:00
Pablo Hoffman 2d91c7136d Merge pull request #464 from kalessin/telnet 
telnet client: fix unexisting reference to engine.slots
 2013-11-19 05:22:58 -08:00
olveyra 755b9ba5a4 telnet client: fix unexisting reference to engine.slots 2013-11-19 04:52:24 +01:00
Pablo Hoffman afe6eaa2fe Merge pull request #460 from tntC4stl3/master 
duplicate 'use' in line 87
 2013-11-15 04:10:49 -08:00
tntC4stl3 b51d5d81e4 duplicate 'use' in line 87 2013-11-15 13:56:44 +08:00
Daniel Graña c74903f9da process_parallel was leaking the failures on its internal deferreds. closes #458 
DeferredList implemented cancellation in Twisted 13.2.0 by holding a
reference to the affected deferreds objects, if a deferred errored the
result was propagated to the DeferredList but still referenced by the
original deferred and nobody was consuming it.

The tests started to fail because the reference from DeferredList
prevented the underlining deferred from been collected before the test
finish invalidating the effect of self.flushedLoggedErrors() call.
 2013-11-09 02:12:52 -02:00
Daniel Graña 04ff7ecebf improve 0.20 release notes 
Conflicts:
	docs/news.rst
 2013-11-08 17:45:03 -02:00