RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,809 Commits 49 Branches 132 Tags 149 MiB
Commit Graph

6809 Commits

Author SHA1 Message Date
Paul Tremberth e42b846a9f Use body to chose response type after decompression content 2017-03-06 23:10:38 +01:00
Mikhail Korobov 451f147468 Merge pull request #2628 from redapple/brotli-docs 
DOC Mention brotli support in HttpCompressionMiddleware section
 2017-03-06 22:16:21 +05:00
Mikhail Korobov 20c4f9f43a Merge pull request #2627 from redapple/referrer-empty-string-header 
Fix referrer policy from response headers and support explicit empty string
 2017-03-06 21:44:08 +05:00
Paul Tremberth 2a7d391e0b DOC Mention brotli support in HttpCompressionMiddleware section 2017-03-06 17:30:32 +01:00
Paul Tremberth 768f3155e5 Fix referrer policy from response headers and support explicit empty string 2017-03-06 16:20:37 +01:00
Paul Tremberth d7b26edf6b Merge pull request #2625 from redapple/release-notes-packaging-fix 
Update release notes for 1.0.7, 1.1.4 and 1.2.3
 2017-03-06 14:47:01 +01:00
Paul Tremberth a8b47d6c68 Update release notes for 1.0.7, 1.1.4 and 1.2.3 2017-03-06 14:25:52 +01:00
Paul Tremberth 0a17f9b55c Merge pull request #2334 from ArturGaspar/data_uri 
[MRG+1] data URI download handler.
 2017-03-06 13:50:46 +01:00
Mikhail Korobov 2c5fa0c130 Merge pull request #2617 from redapple/engine-dupe-statement 
Remove redundant slot.add_request() call in ExecutionEngine
 2017-03-03 21:44:07 +05:00
Paul Tremberth 30d812eea2 Remove redundant slot.add_request() call in ExecutionEngine 2017-03-03 17:15:37 +01:00
Paul Tremberth c68f99eed8 Refactor settings tests 2017-03-03 17:03:25 +01:00
Paul Tremberth c3b6feca0e Fix setting lookup for MEDIA_ALLOWED_REDIRECTS 2017-03-03 16:29:07 +01:00
Konstantin Lopuhin ef04cfd237 Respect log settings in custom_settings: fixes GH-1612 
A new root logger is installed when a crawler is created
if one was already installed before.
This allows to respect custom settings related to logging,
such as LOG_LEVEL, LOG_FILE, etc.
 2017-03-03 18:01:11 +03:00
Paul Tremberth f7e11b198e Cleanup 2017-03-03 16:00:59 +01:00
Paul Tremberth ecde166ee1 Refactor without MEDIA_HTTPSTATUS_LIST setting 2017-03-03 15:52:05 +01:00
Paul Tremberth 72fbb687d7 Revert "expose allowed_status tuple for media pipeline" 
This reverts commit 052809c73ed20b9a728a8fd7df3de5f45f2dad8d.
 2017-03-03 15:52:05 +01:00
Paul Tremberth c64ebee062 Refactor (WIP) 2017-03-03 15:52:05 +01:00
Bernardas 11b31c9fbd fix redirect change 2017-03-03 15:52:05 +01:00
Bernardas 3cef1cd451 adjust variable wording and redirect logic 2017-03-03 15:52:05 +01:00
Bernardas f0b4077f81 expose allowed_status tuple for media pipeline 2017-03-03 15:52:05 +01:00
Bernardas 2e052c8615 fix error when settings are not provided 2017-03-03 15:52:05 +01:00
Bernardas 8542780854 typo and clarify handling 2017-03-03 15:52:05 +01:00
Bernardas 6a42214716 add tests for media pipeline MEDIA_ALLOW_REDIRECTS and MEDIA_HTTPSTATUS_LIST settings 2017-03-03 15:52:05 +01:00
Bernardas 25ed491219 add description for media pipeline MEDIA_ALLOW_REDIRECTS and MEDIA_HTTPSTATUS_LIST settings 2017-03-03 15:52:05 +01:00
Bernardas 5d0058492c add media pipeline settings to enable redirection and handling of certain http statuses 2017-03-03 15:52:05 +01:00
jorenham 5e89db5484 Moved cache dir logging to `open_spider` in FilesystemCacheStorage for consistency 2017-03-03 15:32:20 +01:00
jorenham 42b429dc37 Log full cache file path instead of cache directory for the storages that cache to single files. 2017-03-03 15:15:59 +01:00
Mikhail Korobov 7e8453cf1e Merge pull request #2306 from redapple/referrer-policy 
[MRG] Referrer policies in RefererMiddleware
 2017-03-03 04:05:13 +05:00
Paul Tremberth fad499ab60 Rename arguments (bis) 2017-03-02 23:06:04 +01:00
Paul Tremberth db176f872b Remove Legacy Policy which is equivalent to UnsafeUrl Policy 2017-03-02 22:56:23 +01:00
Mikhail Korobov 93024c242b Merge pull request #2537 from scrapy/no-canonicalize 
[MRG+1] Set canonicalize=False for LinkExtractor
 2017-03-03 02:53:36 +05:00
Paul Tremberth c2c503192f Rename arguments 2017-03-02 22:53:27 +01:00
Artur Gaspar b50d0370f4 Test response attributes in data URI download handler. 2017-03-02 14:46:33 -03:00
jorenham f96490df2c Move cache storage logging to the individual storage classes 2017-03-02 16:17:51 +01:00
Paul Tremberth 12a8ddecab Fix tests 2017-03-02 13:03:18 +01:00
Paul Tremberth e71803c833 Add tests for duplicate spider name warnings 2017-03-02 12:48:47 +01:00
Paul Tremberth 5be5ef57f3 Remove extra blank line 2017-03-02 12:44:40 +01:00
Erick 6abd9ba843 Fix warning to duplicated spider. Issue 2181 2017-03-02 12:44:40 +01:00
MrMenezes f3b75c940d Fix warning to duplicated spider. Issue 2181 2017-03-02 12:44:40 +01:00
jorenham 97d84d920b Logging the cache directory at HttpCacheMiddleware instantiation #2604 2017-03-02 11:04:16 +01:00
Oto Brglez a70ec30e19 Adding new options. 2017-03-01 23:02:42 +01:00
Paul Tremberth 2d55d838ca Fix strip_url() tests 2017-03-01 20:59:52 +01:00
Paul Tremberth efa50039ec Add tests for policy fallback on unknown policies from meta and headers 2017-03-01 17:51:23 +01:00
Paul Tremberth 6916dd6240 Warn or fail with exception on unknown policies 2017-03-01 17:51:23 +01:00
Paul Tremberth 8226e77010 Add test for Referer header on HTTP redirections 2017-03-01 17:51:23 +01:00
Paul Tremberth d2aa51c0fb Update tests 2017-03-01 17:51:23 +01:00
Paul Tremberth 04e4d08612 Pass URLs around instead of Request/Responses 2017-03-01 17:51:23 +01:00
Paul Tremberth bc200d1155 Rename setting to REFERRER_POLICY (with 2 Rs) 2017-03-01 17:51:23 +01:00
Paul Tremberth 537683f945 Add autoclass directives to document built-in policies 2017-03-01 17:51:23 +01:00
Paul Tremberth 3dc09eeceb Use table for referrer policy options 2017-03-01 17:51:23 +01:00