RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,647 Commits 50 Branches 132 Tags 142 MiB
Commit Graph

5647 Commits

Author SHA1 Message Date
Paul Tremberth f14cc821e9 Merge pull request #2113 from scrapy/release-notes-1.1.1 
Update release notes for upcoming 1.1.1 release
 2016-07-13 18:12:56 +02:00
Paul Tremberth 7e7c832f9f Update 1.1.1 release date 2016-07-13 17:59:11 +02:00
Paul Tremberth ed79193f49 Update notes with latest 1.1 commits 2016-07-13 17:49:17 +02:00
Paul Tremberth 51ed24904b Update release notes for upcoming 1.1.1 release 2016-07-13 17:36:48 +02:00
Kenneth Schnall 758802d41b Update practices.rst 2016-07-13 17:25:08 +02:00
Paul Tremberth e28392ca8c Merge pull request #2122 from redapple/backport-1.1-1989 
[backport][1.1] [image_pipeline] bring back uppercase class attributes (PR #1989)
 2016-07-13 17:19:36 +02:00
Pawel Miech 1f8b2c162b [image_pipeline] minor style tweaks 2016-07-13 17:02:13 +02:00
Pawel Miech 44e731898a [image_pipeline] dont use assert in doctest 2016-07-13 17:02:13 +02:00
Pawel Miech d40de1e013 [image_pipeline] style edits 
* 80 characters line limit
* shortening some code
* removed dead code
* add doctest for _key_for_pipe function
 2016-07-13 17:02:13 +02:00
Pawel Miech 0405750117 [FilesPipeline, ImagesPipeline] fix for cls attrs with DEFAULT prefix 
some class attributes for ImagePipeline and FilesPipeline had DEFAULT prefix. These
attributes should be preserved as well, if users subclasses define values for
DEFAULT_<CLS_ATTRIBUTE_NAME> attribute this value should be preserved.
 2016-07-13 17:02:13 +02:00
Pawel Miech d9a9f76ca0 [files-pipeline] update docs with note about settings 
for subclasses.
 2016-07-13 17:02:13 +02:00
Pawel Miech 516618752b [media-pipeline, file-pipeline] allow setting custom settings for subclasses 
* move key_for_pipe function to media pipeline so that file pipeline can use it
* use key_for_pipe in file pipeline so that users can define custom settings for subclasses easily
* add tests for file pipelines attributes and settings
 2016-07-13 17:02:13 +02:00
Pawel Miech 0957f078bd [files_pipeline] ensure class attributes are preserved 
dont override class attributes with default settings (same as in image pipeline).
 2016-07-13 17:02:13 +02:00
Pawel Miech b3591eadaa [files_pipeline] unify tests for files pipeline 
if test tests same thing but for different field it can be unified into one.
 2016-07-13 17:02:13 +02:00
Pawel Miech 20a74dd27a [image_pipeline] another test for subclass inheritance 
test case when subclass inherits from base class and has no attributes nor
settings defined.
 2016-07-13 17:02:13 +02:00
Pawel Miech df4cda79d2 [image_pipeline] refactor and simplify tests for image settings 
unify tests that test same thing for different attribute values into one. Add
better docstrings for tests.
 2016-07-13 17:02:12 +02:00
Pawel Miech dec04d50fa [image_pipeline] unify and simplify tests for setting loading 
there was identical test for different setting keys. I unified it into
one unit test.

Fixes comments for tests, adds comments about intention of uppercase attrs.

Adds another test for user defined setting keys and uppercase attrs.
 2016-07-13 17:02:12 +02:00
Pawel Miech c6277c9963 [image-pipeline] image settings with class name 
allow to have image settings with class name, so that settings for user defined ImagePipeline
subclasses can be defined easily.
 2016-07-13 17:02:12 +02:00
Pawel Miech 6bec0f773b [image_pipeline] tests for class attrs backward compatibility 
and docs about image pipeline settings.
 2016-07-13 17:02:12 +02:00
Pawel Miech 3fce2842ea [image_pipeline] bring back uppercase pipeline attributes 
allow users to have class attributes on image pipelines. This assumes
that class attributes are useful if users want to have different pipeline
classes inhriting from ImagePipeline.
 2016-07-13 17:02:12 +02:00
Paul Tremberth 308da48806 Merge pull request #2121 from redapple/backport-1.1-2094 
[backport][1.1] Catch and ignore certification verification exception for IP-address hosts (PR #2094)
 2016-07-13 16:46:15 +02:00
Paul Tremberth 1b7b87def1 Change wording of warning + docstring for ScrapyClientTLSOptions 2016-07-13 16:24:46 +02:00
Paul Tremberth 498719686c Rephrase warning 2016-07-13 16:24:46 +02:00
Paul Tremberth d2585704b8 Catch and ignore TLS verification exception for IP-address hosts 
Fixes GH-2092
 2016-07-13 16:24:46 +02:00
Paul Tremberth 0774b52245 Merge pull request #2112 from redapple/backport-1.1-pr2065 
[backport][1.1] Do not decompress binary/octet-stream responses (PR #2065)
 2016-07-11 13:37:08 +02:00
Paul Tremberth bda448f17d [HttpCompressionMiddleware] Do not decompress binary/octet-stream responses 2016-07-11 12:37:35 +02:00
Elias Dorneles a40f7c7a86 Merge pull request #2107 from redapple/backport-1.1-pr2052 
[backport][1.1] Enable genspider command outside project folder (PR #2052)
 2016-07-08 14:53:51 -03:00
Elias Dorneles 5b1531b077 Merge pull request #2106 from redapple/backport-1.1-pr2050 
[backport][1.1] Is_gzipped for application/x-gzip;charset=utf-8 (PR #2050)
 2016-07-08 14:53:41 -03:00
Valdir Stumm Junior 848bd4c2a2 document new genspider behavior 2016-07-08 18:48:39 +02:00
Valdir Stumm Junior e239246f4b remove references to Item classes in templates 2016-07-08 18:48:39 +02:00
Valdir Stumm Junior acb0c325aa enable genspider command outside projects 2016-07-08 18:48:39 +02:00
Joakim Uddholm 92bde95874 is_gzipped: Separated tests again. 2016-07-08 18:30:34 +02:00
Joakim Uddholm 658b6a6dcd is_gzipped: improved readability 2016-07-08 18:30:34 +02:00
Joakim Uddholm 27cbcbcdaf is_gzipped: Changed to regex to check the content-type header. Also suggested by @redapple. 2016-07-08 18:30:34 +02:00
Joakim Uddholm 346a376f5b Added new testcases suggested by @redapple. 2016-07-08 18:30:34 +02:00
Joakim Uddholm 5c511a29ec Separated tests based on case 2016-07-08 18:30:34 +02:00
Joakim Uddholm 59fda180d7 Suggested fix for is_gzipped 2016-07-08 18:30:34 +02:00
Joakim Uddholm 1c7ea7d686 Test to show bug with is_gzipped and Content-Type: application/gzip;charset. 2016-07-08 18:30:34 +02:00
Paul Tremberth b809907e0f Merge pull request #2042 from scrapy/backport-1.1-pr1974 
[backport][1.1] Retrying scrapy.core.downloader.handlers.http11.TunnelError (PR #1974)
 2016-07-08 14:11:28 +02:00
Paul Tremberth 3f03f01d80 Merge pull request #2043 from scrapy/backport-1.1-pr2008 
[backport][1.1] HttpCompressionMiddleware: don't decode HEAD responses (PR #2008)
 2016-07-08 14:10:52 +02:00
Paul Tremberth 8c937ed000 Merge pull request #2103 from redapple/backport-1.1-pr2038 
[backport][1.1] Do not fail on canonicalizing URLs with wrong netlocs (PR #2038)
 2016-07-08 12:46:31 +02:00
Paul Tremberth 53be5dc3e0 Do not fail on canonicalizing URLs with wrong netlocs 
Fixes #2010
 2016-07-08 12:35:56 +02:00
Paul Tremberth 0ef490e9ce Merge pull request #2102 from redapple/backport-1.1-pr2026 
[backport][1.1] Encourage descriptive PR titles (PR #2026)
 2016-07-08 12:31:47 +02:00
Νικόλαος-Διγενής Καραγιάννης 8ffb30fbef Encourage descriptive PR titles 2016-07-08 12:30:54 +02:00
Paul Tremberth 060a94b398 Merge pull request #2101 from redapple/backport-1.1-pr2001 
[backport][1.1] Fixed choosing of response class (PR #2001)
 2016-07-08 12:29:33 +02:00
Matvei Nazaruk eb0c1a3921 Added test for http11 choosing response type without content-type header. 2016-07-08 12:20:22 +02:00
Matvei Nazaruk 5e60d957e9 Fixed choosing of response class based on body. 2016-07-08 12:20:21 +02:00
Paul Tremberth 1c34e491d6 Merge pull request #2100 from redapple/backport-1.1-pr2069 
[backport][1.1] Add "Host" header in CONNECT requests to HTTPS proxies (PR #2069)
 2016-07-08 12:14:14 +02:00
Paul Tremberth 556fc2c334 Cleanup unused argument 2016-07-08 11:59:57 +02:00
Paul Tremberth d89f9df52e Fix CONNECT request timeout (with an ugly hack) 2016-07-08 11:59:56 +02:00