RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
9,291 Commits 49 Branches 132 Tags 160 MiB
Commit Graph

9291 Commits

Author SHA1 Message Date
Bernardas 8542780854 typo and clarify handling 2017-03-03 15:52:05 +01:00
Bernardas 6a42214716 add tests for media pipeline MEDIA_ALLOW_REDIRECTS and MEDIA_HTTPSTATUS_LIST settings 2017-03-03 15:52:05 +01:00
Bernardas 25ed491219 add description for media pipeline MEDIA_ALLOW_REDIRECTS and MEDIA_HTTPSTATUS_LIST settings 2017-03-03 15:52:05 +01:00
Bernardas 5d0058492c add media pipeline settings to enable redirection and handling of certain http statuses 2017-03-03 15:52:05 +01:00
jorenham 5e89db5484 Moved cache dir logging to `open_spider` in FilesystemCacheStorage for consistency 2017-03-03 15:32:20 +01:00
jorenham 42b429dc37 Log full cache file path instead of cache directory for the storages that cache to single files. 2017-03-03 15:15:59 +01:00
Mikhail Korobov 7e8453cf1e Merge pull request #2306 from redapple/referrer-policy 
[MRG] Referrer policies in RefererMiddleware
 2017-03-03 04:05:13 +05:00
Paul Tremberth fad499ab60 Rename arguments (bis) 2017-03-02 23:06:04 +01:00
Paul Tremberth db176f872b Remove Legacy Policy which is equivalent to UnsafeUrl Policy 2017-03-02 22:56:23 +01:00
Mikhail Korobov 93024c242b Merge pull request #2537 from scrapy/no-canonicalize 
[MRG+1] Set canonicalize=False for LinkExtractor
 2017-03-03 02:53:36 +05:00
Paul Tremberth c2c503192f Rename arguments 2017-03-02 22:53:27 +01:00
Artur Gaspar b50d0370f4 Test response attributes in data URI download handler. 2017-03-02 14:46:33 -03:00
jorenham f96490df2c Move cache storage logging to the individual storage classes 2017-03-02 16:17:51 +01:00
Paul Tremberth 12a8ddecab Fix tests 2017-03-02 13:03:18 +01:00
Paul Tremberth e71803c833 Add tests for duplicate spider name warnings 2017-03-02 12:48:47 +01:00
Paul Tremberth 5be5ef57f3 Remove extra blank line 2017-03-02 12:44:40 +01:00
Erick 6abd9ba843 Fix warning to duplicated spider. Issue 2181 2017-03-02 12:44:40 +01:00
MrMenezes f3b75c940d Fix warning to duplicated spider. Issue 2181 2017-03-02 12:44:40 +01:00
jorenham 97d84d920b Logging the cache directory at HttpCacheMiddleware instantiation #2604 2017-03-02 11:04:16 +01:00
Oto Brglez a70ec30e19 Adding new options. 2017-03-01 23:02:42 +01:00
Paul Tremberth 2d55d838ca Fix strip_url() tests 2017-03-01 20:59:52 +01:00
Paul Tremberth efa50039ec Add tests for policy fallback on unknown policies from meta and headers 2017-03-01 17:51:23 +01:00
Paul Tremberth 6916dd6240 Warn or fail with exception on unknown policies 2017-03-01 17:51:23 +01:00
Paul Tremberth 8226e77010 Add test for Referer header on HTTP redirections 2017-03-01 17:51:23 +01:00
Paul Tremberth d2aa51c0fb Update tests 2017-03-01 17:51:23 +01:00
Paul Tremberth 04e4d08612 Pass URLs around instead of Request/Responses 2017-03-01 17:51:23 +01:00
Paul Tremberth bc200d1155 Rename setting to REFERRER_POLICY (with 2 Rs) 2017-03-01 17:51:23 +01:00
Paul Tremberth 537683f945 Add autoclass directives to document built-in policies 2017-03-01 17:51:23 +01:00
Paul Tremberth 3dc09eeceb Use table for referrer policy options 2017-03-01 17:51:23 +01:00
Paul Tremberth 605935f015 Edit text 2017-03-01 17:51:23 +01:00
Paul Tremberth eb07285a63 Reword warning on no-referrer-when-downgrade policy 2017-03-01 17:51:23 +01:00
Paul Tremberth 03ff19d188 Update docs for new "referrer_policy" Request.meta key 2017-03-01 17:51:23 +01:00
Paul Tremberth e249abc32b Update docs 2017-03-01 17:50:39 +01:00
Paul Tremberth c86f568b9c Update docs with "strict-..." policies 2017-03-01 17:50:39 +01:00
Paul Tremberth b6c761d2b4 Fix tests 2017-03-01 17:50:39 +01:00
Paul Tremberth ebcacd3f54 Update StrictOriginPolicy 2017-03-01 17:50:39 +01:00
Paul Tremberth deb8567116 Update NoReferrerWhenDowngradePolicy 2017-03-01 17:50:39 +01:00
Paul Tremberth 77aec5a796 Fix implementation 2017-03-01 17:50:39 +01:00
Paul Tremberth 5cef67ae75 Update Referrer tests for "strict-" policies 2017-03-01 17:50:39 +01:00
Paul Tremberth c808a97c74 Add new "strict-" policies 2017-03-01 17:50:39 +01:00
Paul Tremberth 0a0b60a59f Add tests for stripping userinfo with percent-encoded delimiters 2017-03-01 17:50:39 +01:00
Paul Tremberth 8864d0e8c1 Rename helper function to strip_url() + add more tests 2017-03-01 17:50:39 +01:00
Paul Tremberth 5dd7311cd4 Move URL credentials stripping to a helper function 2017-03-01 17:50:39 +01:00
Paul Tremberth c9c59db489 Update documentation about REFERER_POLICY setting 2017-03-01 17:50:39 +01:00
Paul Tremberth 285d5bc03a Patch "Referer" header on HTTP redirects if necessary 2017-03-01 17:50:39 +01:00
Paul Tremberth d3d4d66ce8 Add tests for referrer-policy set in response HTTP headers 2017-03-01 17:50:39 +01:00
Paul Tremberth e50e670eff Add test for custom referrer policy via settings 2017-03-01 17:50:39 +01:00
Paul Tremberth ec8b4c1a9b Change __init__ default "settings" arg handling 2017-03-01 17:50:39 +01:00
Paul Tremberth 0344f57fef Support case-insensitive policy names in settings 2017-03-01 17:50:39 +01:00
Paul Tremberth e72b6e3361 Add tests for referrer policy via settings and via Request meta 2017-03-01 17:50:39 +01:00