RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,240 Commits 49 Branches 132 Tags 179 MiB
Commit Graph

5240 Commits

Author SHA1 Message Date
Nicolás Alejandro Ramírez Quiros 6827eab2c5 Merge pull request #1654 from aron-bordin/show_maxdownsize_warn_once 
[MRG+1] show download warnsize once
 2016-01-19 11:30:54 -03:00
Daniel Graña 49fe631d89 Merge pull request #1676 from lopuhin/py3-webclient 
[MRG+1] py3: fix webclient
 2016-01-19 11:07:54 -03:00
Mikhail Korobov fdc3c9d561 Merge pull request #1624 from side2k/more_verbose_download_cancel 
[MRG+1] added more verbosity for log and for exception when download is cancelled because of a size limit
 2016-01-19 00:17:56 +05:00
Konstantin Lopuhin 4946434582 revert most changes to this test, and clarify - it is valid only on py2, because urls are strictly unicode on py3 2016-01-18 15:23:01 +03:00
Konstantin Lopuhin a2efd389b0 clarify: rename r_transform to response_transform 2016-01-18 15:09:54 +03:00
Elias Dorneles bc1f5353dd Merge pull request #1671 from redapple/xmliter-unicode 
[MRG+1] Support unicode tags in xml iterators (fixes #1665)
 2016-01-18 10:02:18 -02:00
Mikhail Korobov 66f41aba3c Merge pull request #1687 from palego/extra-space 
[MRG+1] fix indentation
 2016-01-18 16:48:42 +05:00
Konstantin Lopuhin f2e2ff5e1f py3: webclient assumes that urls come from Request.url and are ascii-only 2016-01-18 12:20:01 +03:00
Konstantin Lopuhin 673df5e416 add webclient test - check that non-standart body encoding matches Content-Encoding header 2016-01-18 11:44:49 +03:00
Konstantin Lopuhin d3c2b0cf7e py3 webclient: I was mistaken about unicode body, revert conversion to bytes and fix HEAD response 2016-01-18 10:06:14 +03:00
Mikhail Korobov 9ef25d7b68 Merge pull request #1686 from cclauss/patch-1 
Simplify if statement
 2016-01-18 11:57:06 +05:00
cclauss cd735e377c Simplify if statement 2016-01-18 07:45:36 +01:00
Paul Tremberth 4e44766653 Use "unicode" string for lxml.etree.tostring() serialization 2016-01-15 19:51:21 +01:00
Elias Dorneles 3f1f15bc4d Merge pull request #1683 from rgtk/patch-2 
Update Stats Collection documentation for @master
 2016-01-15 16:30:52 -02:00
Ralph Gutkowski 79147a61a7 Update stats.rst 2016-01-15 19:25:56 +01:00
Ralph Gutkowski bb38400db5 Update Stats Collection documentation 
`pages_crawled` value doesn't exist. Replace it with `downloader/response_count` in order to avoid confusion.
 2016-01-15 19:00:58 +01:00
Konstantin Lopuhin 5c2241ccc7 py3: fix webclient tests after making ScrapyHTTPClientFactory use bytes as in twisted 2016-01-14 15:30:28 +03:00
Konstantin Lopuhin 8df35bcac6 rm note to self: to be discussed in PR 2016-01-14 14:52:44 +03:00
Konstantin Lopuhin ac2cf191d1 py3: remove comments, utf-8 is fine here: as twisted ultimately uses urllib.parse.quote that assepts bytes and assumes utf-8 2016-01-14 14:50:58 +03:00
Konstantin Lopuhin e5fb609438 make ScrapyHTTPClientFactory comply to twisted HTTPClientFactory protocol - use bytes (encoding are likely wrong at this stage) 2016-01-14 14:45:02 +03:00
Konstantin Lopuhin 6a412d2503 all tests pass in test_webclient.py on py3 - removing from py3-ignores 2016-01-14 13:48:48 +03:00
Konstantin Lopuhin 0178356178 py3 fix testRedirect: url is bytes here 2016-01-14 13:46:46 +03:00
Konstantin Lopuhin 30c7b4e4cc py3 compat in test_timeoutTriggering cleanup 2016-01-14 13:44:14 +03:00
Konstantin Lopuhin 88f55312af py3 fix in testFactoryInfo - factory attirbutes are bytes in twisted 2016-01-14 13:43:37 +03:00
Konstantin Lopuhin b5f9bc8499 py3 test fixes in test_webclient - expect bytes as page body 2016-01-14 13:04:45 +03:00
Konstantin Lopuhin ae4aa2c3b2 py3 test fix: putChild expects bytes as path 2016-01-14 13:04:10 +03:00
Konstantin Lopuhin 325b6af6c2 fix ScrapyHTTPPageGetterTests for py3 - we expect bytes here 2016-01-14 12:29:59 +03:00
Konstantin Lopuhin 945674eb8f pass test_externalUnicodeInterference - the logic for py3 is clearly inverse of what was expected in this test, as scrapy Request url must be unicode 2016-01-14 12:25:54 +03:00
Konstantin Lopuhin 1d5ab67183 pass test_getPageHead on py3 2016-01-14 12:04:26 +03:00
Konstantin Lopuhin 73ff87c1dc decode body from utf-8, as scrapy stores body as bytes, and twisted has already converted to unicode 2016-01-14 12:03:08 +03:00
Konstantin Lopuhin 9f2be23a39 webclient tests, py3: fix setUp, pass test_getPage 2016-01-14 11:42:23 +03:00
Konstantin Lopuhin f3889b0bce py3 compat: encode delimiter, method and path in ScrapyHTTPPageGetter 2016-01-14 11:41:49 +03:00
Paul Tremberth 1347015a80 Refactored test code 2016-01-13 12:32:28 +01:00
Paul Tremberth d4c7d72b2b Add tests for input type in xmliter calls 2016-01-13 12:13:47 +01:00
Elias Dorneles 57f99fc34e Merge pull request #1644 from yarikoptic/master 
[MRG+1] BF: robustify _monkeypatches check for twisted - str() name first (Closes #1634)
 2016-01-13 08:59:48 -02:00
Paul Tremberth 9fad25f3d1 Use explicit Unicode and bytes for XML body in tests 2016-01-13 11:42:41 +01:00
Paul Tremberth 2f2c2e8096 Merge remote-tracking branch 'origin/master' into xmliter-unicode 2016-01-13 11:42:01 +01:00
Mikhail Korobov f01fd07642 Merge pull request #1661 from redapple/py3-iterators 
[MRG+1] PY3: port utils/iterators
 2016-01-13 10:03:02 +05:00
Paul Tremberth d7d4ef67a6 Changes following comments 2016-01-12 11:08:49 +01:00
Paul Tremberth 6ddd814738 Support unicode tags in xml iterators (fixes #1665) 2016-01-12 10:48:45 +01:00
Paul Tremberth d4872940db PY3: port utils/iterators 2016-01-07 10:51:27 +01:00
Elias Dorneles 95e8ff8ba1 Merge pull request #1660 from stummjr/patch-4 
[MRG+1] Update deprecated examples
 2016-01-06 10:43:08 -02:00
Valdir Stumm Jr 2abc9bc901 Update deprecated examples 
* update the scrapy.org example to deal with the new layout.
* replaced slashdot.org by reddit.com, because it seems that slashdot is blocking requests.
 2016-01-06 10:29:45 -02:00
palego d03d262f52 indentation 2016-01-04 10:00:13 +01:00
Daniel Graña 1b435b2887 Add 1.0.4 release notes 2015-12-30 15:43:54 -03:00
Aron Bordin f57121c77b show download warnsize once 2015-12-30 13:10:13 -02:00
Yaroslav Halchenko 4f49aab7c0 BF: robustify _monkeypatches check for twisted - str() name first (Closes: #1634) 
In my case, while running datalad tests using nose, scrapy was failing
since v was None
 2015-12-18 16:16:05 -05:00
Mikhail Korobov a5db7f862c Merge pull request #1639 from seales/master 
Spelling fixes
 2015-12-14 23:33:14 +05:00
Mikhail Korobov 1bc243a876 Merge pull request #1638 from orangain/toc-captions 
[MRG+1] DOC: Add captions to toctrees which appear in sidebar
 2015-12-14 23:32:18 +05:00
seales 719b1353a7 Spelling fixes 2015-12-13 19:39:48 -08:00