RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,450 Commits 49 Branches 132 Tags 132 MiB
Commit Graph

709 Commits

Author SHA1 Message Date
Paul Tremberth 15a5c533fa Add tests for HTTP 307 permanent redirects 2017-07-26 19:07:57 +02:00
Paul Tremberth 1fdc10684f HTTP Cache: treat 308 as 301 2017-07-26 19:01:48 +02:00
Paul Tremberth 5dc9a88c34 Handle HTTP 308 Permanent Redirect 2017-07-26 19:01:48 +02:00
Daniel Graña 2371a2a0df Merge pull request #2789 from starrify/add-response-follow-tag-link 
[MRG+1] Added: Now supporting <link> tags in Response.follow
 2017-07-24 15:41:30 -03:00
Daniel Graña 45b81693b4 Merge pull request #2769 from stummjr/issue-2766 
[MRG+1] Add verification to check if Request callback is callable
 2017-07-24 15:14:33 -03:00
Paul Tremberth 1f08d9a648 Add test for DNS cache disabling 2017-07-04 23:10:19 +02:00
Konstantin Lopuhin 7c67047e77 Fix get_func_args tests under PyPy 
On CPython get_func_args does not work correctly for built-in
methods.
 2017-06-15 13:07:59 +03:00
Konstantin Lopuhin 5abb70c8d7 Fix test_weakkeycache on PyPy: run gc.collect() 
One gc.collect() seems to be enough, but it's more reliable
to run it several times (at most 100), until all objects are collected.
 2017-06-15 13:06:30 +03:00
Konstantin Lopuhin c3d17659b3 Fix queue serialization test on PyPy 
It is not affected by Twisted bug #7989 and is more permissive
with pickling (especially with protocol=2).
 2017-06-15 13:05:45 +03:00
Konstantin Lopuhin 6014856df5 Fix test_output_processor_error undere PyPy 
For float(u'$10') PyPy includes "u'" in the error message,
and it's more fair to check error message on input we are really
passing.
 2017-06-15 12:57:32 +03:00
Pengyu CHEN b33e0d5a54 Added: Now supporting <link> tags in Response.follow 2017-06-14 17:09:42 +08:00
Valdir Stumm Junior 4b6f68b9ee make reqser tests create Request with proper callback/errback 2017-06-05 17:26:52 -03:00
Valdir Stumm Junior 60727dedf6 verify if Request callback is callable 2017-05-31 15:00:38 -03:00
Daniel Graña 25f609e2a3 Merge pull request #2675 from simongartz/png-p-to-jpg-conversion-fix 
[MRG+1] Fixes conversion of transparent PNG with palette images to jpg #2452
 2017-05-18 16:34:02 -03:00
Mikhail Korobov 532400f993 Merge pull request #2643 from harshasrinivas/set-retry-times-per-request 
[MRG+1] Add feature to set RETRY_TIMES per request (#2642)
 2017-05-17 15:46:33 +05:00
Daniel Graña 73668ce407 Merge pull request #2721 from HarrisonGregg/feature-drop-from-response-field 
[MRG+1] Allow dropping field in from_response formdata
 2017-05-16 09:47:45 -03:00
Harrison Gregg ffef828a8d Add test for dropping fields in from_response request body 2017-05-15 12:25:32 -04:00
Paul Tremberth dfe6d3d59a Merge pull request #2456 from elacuesta/feed_export_beautify 
[MRG+1] Feed exports: beautify JSON and XML
 2017-05-12 18:12:48 +02:00
Eugenio Lacuesta 63b8caf5de Feed exports: rewrite indentation test without .strip() 2017-05-09 11:58:53 -03:00
Daniel Graña 362d6f2d20 Merge pull request #2622 from rolando-contrib/download-maxsize-abort 
[MRG+1] Abort connection earlier and avoid to buffer data when max size limit is reached
 2017-05-04 11:44:51 -03:00
Paul Tremberth 6d14e392f1 Remove old test certificate+key 2017-04-27 23:35:01 +02:00
Paul Tremberth e6ab8bc9a5 Change "localhost" test server certificate 2017-04-27 23:22:25 +02:00
Rolando Espinoza 2b34c6edff Abort connection earlier and avoid to buffer data 
A symptom of this issue was having the log message "Received (X) bytes
larger than download max size (Y)" several times printed, with increased
X values.
 2017-04-27 13:28:16 -03:00
Paul Tremberth c3d0f9b6c1 Add test for non-duplicated `Content-Length: 0` for bodyless POST 2017-04-25 17:03:41 +02:00
Paul Tremberth b1a0a6e258 Make mockserver runnable outside of tox 
Add POST support for Echo resource
 2017-04-25 17:01:54 +02:00
harshasrinivas 38e6857c95 Improvise the clarity of test cases 2017-03-23 19:45:04 +05:30
Simon Diviani Gartz 21d794d35a Fixes conversion of transparent PNG with palette images to jpg #2452 2017-03-22 13:29:15 +01:00
Paul Tremberth 776129a951 Merge pull request #2649 from pawelmhm/logformatter-2647 
[MRG+2] [logformatter] 'flags' format spec backward compatibility
 2017-03-21 10:51:31 +01:00
harshasrinivas 0d9ebd6e1e Update tests for max_retry_times 2017-03-19 06:15:46 +05:30
Pawel Miech 4345eaf1b6 [logformatter] backward compat comments 2017-03-17 08:11:20 +01:00
Mikhail Korobov ec55799d5e Merge pull request #2616 from redapple/mediapipeline-redirect-fix-continued 
[MRG] Allow redirections in media files downloads
 2017-03-16 15:31:28 +05:00
woxcab a84652e775 Init tests are split by initializer' input 2017-03-15 12:39:48 +03:00
harshasrinivas e321ac9931 Update unittests for max_retry_times 2017-03-15 04:12:32 +05:30
harshasrinivas 966bd49c42 Update unittest for meta['max_retry_times'] 2017-03-14 16:23:47 +05:30
Pawel Miech 0f2a5cdb8e [logformatter] 'flags' format spec backward compatibility 
pass 'flags' kwarg to logger so that it is compatible with old
format of CRAWLEDMSG.
 2017-03-13 15:16:38 +01:00
woxcab fbb411a805 Allowed passing objects of Mapping class or its subclass to the CaselessDict initializer 2017-03-13 14:16:39 +03:00
Paul Tremberth 871134ee22 Refactor to also test FilesPipeline 2017-03-12 17:30:24 +01:00
Paul Tremberth 708f1b009b Add integration tests for MEDIA_ALLOW_REDIRECTS 2017-03-10 21:36:33 +01:00
Paul Tremberth 7dcc86e61a Add file listing resource + redirecting resource to MockServer 2017-03-10 21:35:25 +01:00
Mikhail Korobov 9c69e90056 Merge pull request #2632 from redapple/spider-loader-warn-or-fail 
[MRG] Add SPIDER_LOADER_WARN_ONLY to toggle between spiderloader failure or warning
 2017-03-09 23:01:27 +05:00
Paul Tremberth c0cbaccb7b Merge pull request #2581 from lopuhin/respect-custom-log-level 
[MRG+1] Respect custom log level: fixes GH-1612
 2017-03-09 12:47:20 +01:00
Paul Tremberth 9cfe9ae098 Do not use self.assertRaises() as context manager 2017-03-09 12:21:03 +01:00
Paul Tremberth 7be773e14a Add SPIDER_LOADER_WARN_ONLY to toggle between spiderloader failure and warning 2017-03-07 17:40:40 +01:00
Mikhail Korobov d3f8f3d38a Merge pull request #2612 from redapple/dupe-spider-name-tests 
[WIP] Add warning on duplicate spider name
 2017-03-07 20:08:42 +05:00
Eugenio Lacuesta c7bb2fa8ce Feed exports: consistent and backwards compatible behaviour on indent 2017-03-07 11:56:00 -03:00
Eugenio Lacuesta 766b2c8453 Feed exports: enforce difference between None and 0 on indent 
Also rename params and settings from "indent_width" to just "indent"
 2017-03-07 11:56:00 -03:00
Eugenio Lacuesta 7e9153b38d Feed exports: beautify JSON and XML 2017-03-07 11:56:00 -03:00
Paul Tremberth 0b9a18e1a1 Warn only once for all spiders 2017-03-07 15:41:17 +01:00
Paul Tremberth 978306a223 Fix dupe spider name warning string tests 2017-03-07 14:48:16 +01:00
Paul Tremberth b6378c7ef6 Revert to using self.assert methods 2017-03-07 12:28:24 +01:00