RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,265 Commits 49 Branches 132 Tags 170 MiB
Commit Graph

5265 Commits

Author SHA1 Message Date
Konstantin Lopuhin 076ef5537e Merge branch 'master' into py3-http-downloaders 2016-01-19 17:51:42 +03:00
Konstantin Lopuhin 07577c432e Merge branch 'master' into py3-http-downloaders 2016-01-19 17:39:22 +03:00
Konstantin Lopuhin 9c3117a914 more pythonic check of noconnect in proxy params 2016-01-19 17:32:53 +03:00
Konstantin Lopuhin bb50c0be2f remove unused import 2016-01-19 17:30:59 +03:00
Nicolás Alejandro Ramírez Quiros 6827eab2c5 Merge pull request #1654 from aron-bordin/show_maxdownsize_warn_once 
[MRG+1] show download warnsize once
 2016-01-19 11:30:54 -03:00
Daniel Graña 49fe631d89 Merge pull request #1676 from lopuhin/py3-webclient 
[MRG+1] py3: fix webclient
 2016-01-19 11:07:54 -03:00
Mikhail Korobov fdc3c9d561 Merge pull request #1624 from side2k/more_verbose_download_cancel 
[MRG+1] added more verbosity for log and for exception when download is cancelled because of a size limit
 2016-01-19 00:17:56 +05:00
Konstantin Lopuhin de98d8d006 move comment about test skipped on py3 into a proper place 2016-01-18 19:27:31 +03:00
Konstantin Lopuhin 7fdd3225b2 fix test skipping logic - this is (temporary) py2-only part 2016-01-18 19:09:09 +03:00
Konstantin Lopuhin 120fb4adeb revert bogus change 2016-01-18 19:00:40 +03:00
Konstantin Lopuhin f876654a0c Merge branch 'py3-webclient' into py3-http-downloaders 2016-01-18 15:50:58 +03:00
Konstantin Lopuhin 4946434582 revert most changes to this test, and clarify - it is valid only on py2, because urls are strictly unicode on py3 2016-01-18 15:23:01 +03:00
Konstantin Lopuhin a2efd389b0 clarify: rename r_transform to response_transform 2016-01-18 15:09:54 +03:00
Konstantin Lopuhin b940606b7e this is a test for TunnelingTCP4ClientEndpoint - move into Http11ProxyTestCase 2016-01-18 15:06:15 +03:00
Elias Dorneles bc1f5353dd Merge pull request #1671 from redapple/xmliter-unicode 
[MRG+1] Support unicode tags in xml iterators (fixes #1665)
 2016-01-18 10:02:18 -02:00
Konstantin Lopuhin 7af64e8fd2 py3: remove extra encoding/decoding of url: pass it as bytes only when required 2016-01-18 15:00:43 +03:00
Mikhail Korobov 66f41aba3c Merge pull request #1687 from palego/extra-space 
[MRG+1] fix indentation
 2016-01-18 16:48:42 +05:00
Konstantin Lopuhin 0f527849f2 https proxy tunneling - add a test (not perfect, but covers all impl) and fix for py3 2016-01-18 14:44:04 +03:00
Konstantin Lopuhin 98c060d0b2 py3: fix http 1.1 test with https, and use self.host everywhere 2016-01-18 12:42:21 +03:00
Konstantin Lopuhin 04f69fd184 add https 1.1 downloader test - localhost is a valid DNS-ID 2016-01-18 12:37:46 +03:00
Konstantin Lopuhin f403c88053 Merge branch 'py3-webclient' into py3-http-downloaders 2016-01-18 12:25:51 +03:00
Konstantin Lopuhin f2e2ff5e1f py3: webclient assumes that urls come from Request.url and are ascii-only 2016-01-18 12:20:01 +03:00
Konstantin Lopuhin 673df5e416 add webclient test - check that non-standart body encoding matches Content-Encoding header 2016-01-18 11:44:49 +03:00
Konstantin Lopuhin d3c2b0cf7e py3 webclient: I was mistaken about unicode body, revert conversion to bytes and fix HEAD response 2016-01-18 10:06:14 +03:00
Mikhail Korobov 9ef25d7b68 Merge pull request #1686 from cclauss/patch-1 
Simplify if statement
 2016-01-18 11:57:06 +05:00
cclauss cd735e377c Simplify if statement 2016-01-18 07:45:36 +01:00
Paul Tremberth 4e44766653 Use "unicode" string for lxml.etree.tostring() serialization 2016-01-15 19:51:21 +01:00
Elias Dorneles 3f1f15bc4d Merge pull request #1683 from rgtk/patch-2 
Update Stats Collection documentation for @master
 2016-01-15 16:30:52 -02:00
Ralph Gutkowski 79147a61a7 Update stats.rst 2016-01-15 19:25:56 +01:00
Ralph Gutkowski bb38400db5 Update Stats Collection documentation 
`pages_crawled` value doesn't exist. Replace it with `downloader/response_count` in order to avoid confusion.
 2016-01-15 19:00:58 +01:00
Konstantin Lopuhin a4ca1668d8 add https test for http10 handler (no luck with testing https with http11 so far) 2016-01-15 14:20:19 +03:00
Konstantin Lopuhin 96fcf4cea4 add a check that byte url is not accepted in http.Response on py3 2016-01-15 13:27:28 +03:00
Konstantin Lopuhin 99f1f2ad1d unskip tests and modules ported to py3 2016-01-14 19:00:48 +03:00
Konstantin Lopuhin 81a90c3af6 unskip part of test_download_gzip_response on py3, file a twisted issue for the remaining part 2016-01-14 18:47:06 +03:00
Konstantin Lopuhin 2aa6c92ffc py3 fixes in tests.mockserver 2016-01-14 17:52:50 +03:00
Konstantin Lopuhin f46a9d595d skip ftp tests on py3 for now 2016-01-14 17:31:58 +03:00
Konstantin Lopuhin 4950f5988e py3: pass http proxy tests 2016-01-14 17:24:08 +03:00
Konstantin Lopuhin c6f14a39de py3: fix http10 downloader - unicode host expected here 2016-01-14 16:50:16 +03:00
Konstantin Lopuhin 6b79fffa9a py3: pass all of HttpTestCase 2016-01-14 16:37:18 +03:00
Konstantin Lopuhin 3509378b8b py3: pass first http downloader test, simple crawler works now, yay! 2016-01-14 16:29:19 +03:00
Konstantin Lopuhin 32bb5b682a fix import of test_downloader_handlers.py: use @implementer, move failing on py3 imports into corresponding tests 2016-01-14 16:11:16 +03:00
Konstantin Lopuhin 5c2241ccc7 py3: fix webclient tests after making ScrapyHTTPClientFactory use bytes as in twisted 2016-01-14 15:30:28 +03:00
Konstantin Lopuhin 8df35bcac6 rm note to self: to be discussed in PR 2016-01-14 14:52:44 +03:00
Konstantin Lopuhin ac2cf191d1 py3: remove comments, utf-8 is fine here: as twisted ultimately uses urllib.parse.quote that assepts bytes and assumes utf-8 2016-01-14 14:50:58 +03:00
Konstantin Lopuhin e5fb609438 make ScrapyHTTPClientFactory comply to twisted HTTPClientFactory protocol - use bytes (encoding are likely wrong at this stage) 2016-01-14 14:45:02 +03:00
Konstantin Lopuhin 6a412d2503 all tests pass in test_webclient.py on py3 - removing from py3-ignores 2016-01-14 13:48:48 +03:00
Konstantin Lopuhin 0178356178 py3 fix testRedirect: url is bytes here 2016-01-14 13:46:46 +03:00
Konstantin Lopuhin 30c7b4e4cc py3 compat in test_timeoutTriggering cleanup 2016-01-14 13:44:14 +03:00
Konstantin Lopuhin 88f55312af py3 fix in testFactoryInfo - factory attirbutes are bytes in twisted 2016-01-14 13:43:37 +03:00
Konstantin Lopuhin b5f9bc8499 py3 test fixes in test_webclient - expect bytes as page body 2016-01-14 13:04:45 +03:00