RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,505 Commits 49 Branches 132 Tags 162 MiB
Commit Graph

5505 Commits

Author SHA1 Message Date
Elias Dorneles e8b26e2ab2 Merge pull request #1698 from carlosp420/typo-you 
fixed typo You -> you
 2016-01-19 14:18:43 -02:00
carlosp420 e15f361b05 fixed typo You -> you 2016-01-19 11:12:43 -05:00
Elias Dorneles b4fb9d3534 Merge pull request #1678 from lopuhin/py3-http-downloaders 
[MRG+1] Py3: port http downloaders
 2016-01-19 14:10:00 -02:00
Paul Tremberth 2b5245839c Remove unused import statement 2016-01-19 17:04:57 +01:00
Paul Tremberth 1f2233837a Use if Py2/Py3 function instead of custom GzipFile class method 2016-01-19 16:58:24 +01:00
Konstantin Lopuhin 4c84172840 move leveldb to tests/requirements-py3.txt 2016-01-19 18:24:58 +03:00
Konstantin Lopuhin 076ef5537e Merge branch 'master' into py3-http-downloaders 2016-01-19 17:51:42 +03:00
Konstantin Lopuhin 07577c432e Merge branch 'master' into py3-http-downloaders 2016-01-19 17:39:22 +03:00
Konstantin Lopuhin 9c3117a914 more pythonic check of noconnect in proxy params 2016-01-19 17:32:53 +03:00
Konstantin Lopuhin bb50c0be2f remove unused import 2016-01-19 17:30:59 +03:00
Nicolás Alejandro Ramírez Quiros 6827eab2c5 Merge pull request #1654 from aron-bordin/show_maxdownsize_warn_once 
[MRG+1] show download warnsize once
 2016-01-19 11:30:54 -03:00
Konstantin Lopuhin 5ac54ed339 raise minimal twisted version for py3 2016-01-19 17:25:50 +03:00
Daniel Graña 49fe631d89 Merge pull request #1676 from lopuhin/py3-webclient 
[MRG+1] py3: fix webclient
 2016-01-19 11:07:54 -03:00
Mikhail Korobov fdc3c9d561 Merge pull request #1624 from side2k/more_verbose_download_cancel 
[MRG+1] added more verbosity for log and for exception when download is cancelled because of a size limit
 2016-01-19 00:17:56 +05:00
Mikhail Korobov ff235fa19a Remove --lsprof command-line option. Fixes GH-1531 2016-01-19 00:00:31 +05:00
Paul Tremberth fd99ef86df Test for AttributeError when pickling objects (Python>=3.5) 
Same "fix" as in e.g. https://github.com/joblib/joblib/pull/246
 2016-01-18 18:14:33 +01:00
Konstantin Lopuhin de98d8d006 move comment about test skipped on py3 into a proper place 2016-01-18 19:27:31 +03:00
Konstantin Lopuhin 7ffcc58759 Merge branch 'py3-http-downloaders' into py3-downloader-middleware 2016-01-18 19:23:12 +03:00
Konstantin Lopuhin 7fdd3225b2 fix test skipping logic - this is (temporary) py2-only part 2016-01-18 19:09:09 +03:00
Konstantin Lopuhin 120fb4adeb revert bogus change 2016-01-18 19:00:40 +03:00
Paul Tremberth bcbad2905d Stick with ValueError for queue/serialization exception tests 2016-01-18 15:22:29 +01:00
Konstantin Lopuhin 324f2c1a6f common test_downloadermiddleware.py also passes now due to fixes in compression middleware 2016-01-18 16:45:22 +03:00
Konstantin Lopuhin 0b9336418e py3: port compression downloader middleware and tests 2016-01-18 16:43:58 +03:00
Konstantin Lopuhin 6edd4dec33 Merge branch 'py3-http-downloaders' into py3-downloader-middleware 2016-01-18 15:51:26 +03:00
Konstantin Lopuhin f876654a0c Merge branch 'py3-webclient' into py3-http-downloaders 2016-01-18 15:50:58 +03:00
Konstantin Lopuhin 4946434582 revert most changes to this test, and clarify - it is valid only on py2, because urls are strictly unicode on py3 2016-01-18 15:23:01 +03:00
Konstantin Lopuhin a2efd389b0 clarify: rename r_transform to response_transform 2016-01-18 15:09:54 +03:00
Konstantin Lopuhin b940606b7e this is a test for TunnelingTCP4ClientEndpoint - move into Http11ProxyTestCase 2016-01-18 15:06:15 +03:00
Elias Dorneles bc1f5353dd Merge pull request #1671 from redapple/xmliter-unicode 
[MRG+1] Support unicode tags in xml iterators (fixes #1665)
 2016-01-18 10:02:18 -02:00
Konstantin Lopuhin 7af64e8fd2 py3: remove extra encoding/decoding of url: pass it as bytes only when required 2016-01-18 15:00:43 +03:00
Mikhail Korobov 66f41aba3c Merge pull request #1687 from palego/extra-space 
[MRG+1] fix indentation
 2016-01-18 16:48:42 +05:00
Konstantin Lopuhin 0f527849f2 https proxy tunneling - add a test (not perfect, but covers all impl) and fix for py3 2016-01-18 14:44:04 +03:00
Konstantin Lopuhin 98c060d0b2 py3: fix http 1.1 test with https, and use self.host everywhere 2016-01-18 12:42:21 +03:00
Konstantin Lopuhin 04f69fd184 add https 1.1 downloader test - localhost is a valid DNS-ID 2016-01-18 12:37:46 +03:00
Konstantin Lopuhin f403c88053 Merge branch 'py3-webclient' into py3-http-downloaders 2016-01-18 12:25:51 +03:00
Konstantin Lopuhin f2e2ff5e1f py3: webclient assumes that urls come from Request.url and are ascii-only 2016-01-18 12:20:01 +03:00
palego 6b905a9aec split-up the assertIn 
to deal with OS X intricacies (directories prefixed with /private)
 2016-01-18 10:18:31 +01:00
Konstantin Lopuhin 673df5e416 add webclient test - check that non-standart body encoding matches Content-Encoding header 2016-01-18 11:44:49 +03:00
Konstantin Lopuhin d3c2b0cf7e py3 webclient: I was mistaken about unicode body, revert conversion to bytes and fix HEAD response 2016-01-18 10:06:14 +03:00
Mikhail Korobov 9ef25d7b68 Merge pull request #1686 from cclauss/patch-1 
Simplify if statement
 2016-01-18 11:57:06 +05:00
cclauss cd735e377c Simplify if statement 2016-01-18 07:45:36 +01:00
Paul Tremberth 4e44766653 Use "unicode" string for lxml.etree.tostring() serialization 2016-01-15 19:51:21 +01:00
Elias Dorneles 3f1f15bc4d Merge pull request #1683 from rgtk/patch-2 
Update Stats Collection documentation for @master
 2016-01-15 16:30:52 -02:00
Ralph Gutkowski 79147a61a7 Update stats.rst 2016-01-15 19:25:56 +01:00
Ralph Gutkowski bb38400db5 Update Stats Collection documentation 
`pages_crawled` value doesn't exist. Replace it with `downloader/response_count` in order to avoid confusion.
 2016-01-15 19:00:58 +01:00
Paul Tremberth 73d78ec99f Add Code of Conduct Version 1.3.0 from http://contributor-covenant.org/ 
Closes #1645
 2016-01-15 18:01:04 +01:00
Paul Tremberth ee4fadc007 Use .read1() if available when using GzipFile 2016-01-15 14:57:15 +01:00
Konstantin Lopuhin 7d44c5dcea py3: unskip tests/test_downloadermiddleware_httpcache.py and scrapy/downloadermiddlewares/httpcache.py 2016-01-15 15:41:31 +03:00
Konstantin Lopuhin 085fdd6283 py3 fix for ignoring cache controls - map is not a list 2016-01-15 15:40:45 +03:00
Konstantin Lopuhin b0648271d6 py3 fix for rfc1123_to_epoch - "except Exception" was hiding bytes/str error 2016-01-15 15:17:30 +03:00