RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,205 Commits 49 Branches 132 Tags 182 MiB
Commit Graph

6205 Commits

Author SHA1 Message Date
Paul Tremberth efa50039ec Add tests for policy fallback on unknown policies from meta and headers 2017-03-01 17:51:23 +01:00
Paul Tremberth 6916dd6240 Warn or fail with exception on unknown policies 2017-03-01 17:51:23 +01:00
Paul Tremberth 8226e77010 Add test for Referer header on HTTP redirections 2017-03-01 17:51:23 +01:00
Paul Tremberth d2aa51c0fb Update tests 2017-03-01 17:51:23 +01:00
Paul Tremberth 04e4d08612 Pass URLs around instead of Request/Responses 2017-03-01 17:51:23 +01:00
Paul Tremberth bc200d1155 Rename setting to REFERRER_POLICY (with 2 Rs) 2017-03-01 17:51:23 +01:00
Paul Tremberth 537683f945 Add autoclass directives to document built-in policies 2017-03-01 17:51:23 +01:00
Paul Tremberth 3dc09eeceb Use table for referrer policy options 2017-03-01 17:51:23 +01:00
Paul Tremberth 605935f015 Edit text 2017-03-01 17:51:23 +01:00
Paul Tremberth eb07285a63 Reword warning on no-referrer-when-downgrade policy 2017-03-01 17:51:23 +01:00
Paul Tremberth 03ff19d188 Update docs for new "referrer_policy" Request.meta key 2017-03-01 17:51:23 +01:00
Paul Tremberth e249abc32b Update docs 2017-03-01 17:50:39 +01:00
Paul Tremberth c86f568b9c Update docs with "strict-..." policies 2017-03-01 17:50:39 +01:00
Paul Tremberth b6c761d2b4 Fix tests 2017-03-01 17:50:39 +01:00
Paul Tremberth ebcacd3f54 Update StrictOriginPolicy 2017-03-01 17:50:39 +01:00
Paul Tremberth deb8567116 Update NoReferrerWhenDowngradePolicy 2017-03-01 17:50:39 +01:00
Paul Tremberth 77aec5a796 Fix implementation 2017-03-01 17:50:39 +01:00
Paul Tremberth 5cef67ae75 Update Referrer tests for "strict-" policies 2017-03-01 17:50:39 +01:00
Paul Tremberth c808a97c74 Add new "strict-" policies 2017-03-01 17:50:39 +01:00
Paul Tremberth 0a0b60a59f Add tests for stripping userinfo with percent-encoded delimiters 2017-03-01 17:50:39 +01:00
Paul Tremberth 8864d0e8c1 Rename helper function to strip_url() + add more tests 2017-03-01 17:50:39 +01:00
Paul Tremberth 5dd7311cd4 Move URL credentials stripping to a helper function 2017-03-01 17:50:39 +01:00
Paul Tremberth c9c59db489 Update documentation about REFERER_POLICY setting 2017-03-01 17:50:39 +01:00
Paul Tremberth 285d5bc03a Patch "Referer" header on HTTP redirects if necessary 2017-03-01 17:50:39 +01:00
Paul Tremberth d3d4d66ce8 Add tests for referrer-policy set in response HTTP headers 2017-03-01 17:50:39 +01:00
Paul Tremberth e50e670eff Add test for custom referrer policy via settings 2017-03-01 17:50:39 +01:00
Paul Tremberth ec8b4c1a9b Change __init__ default "settings" arg handling 2017-03-01 17:50:39 +01:00
Paul Tremberth 0344f57fef Support case-insensitive policy names in settings 2017-03-01 17:50:39 +01:00
Paul Tremberth e72b6e3361 Add tests for referrer policy via settings and via Request meta 2017-03-01 17:50:39 +01:00
Paul Tremberth 842ce131aa Make default referrer policy customizable via settings 2017-03-01 17:50:39 +01:00
Paul Tremberth f6205778f3 Refactor ReferrerPolicy methods 2017-03-01 17:50:39 +01:00
Paul Tremberth f6a800fde6 Remove all non-cached urlparsing references 2017-03-01 17:50:39 +01:00
Paul Tremberth 59cb884ace Use urlparse_cached() for OriginWhenCrossOriginPolicy 2017-03-01 17:50:39 +01:00
Paul Tremberth f2ee6be3bb Use urlparse_cached() for OriginPolicy 2017-03-01 17:50:39 +01:00
Paul Tremberth 3af88a2877 Use urlparse_cached() on request and responses 2017-03-01 17:50:39 +01:00
Paul Tremberth 7ec1b5f6c3 Add tests for the different referrer policies 2017-03-01 17:50:38 +01:00
Paul Tremberth baed7c436f WIP Add Referrer policies 2017-03-01 17:50:38 +01:00
Mikhail Korobov 7b49b9c0f5 Merge pull request #2590 from rolando-contrib/handle-data-loss-gracefully 
[MRG+2] Handle data loss gracefully.
 2017-03-01 20:23:19 +05:00
Rolando Espinoza f01ae6ffcd Handle data loss gracefully. 
Websites that return a wrong ``Content-Length`` header may cause a data
loss error. Also when a chunked response is not finished properly.

This change adds a new setting ``DOWNLOAD_FAIL_ON_DATALOSS`` (default:
``True``) and request.meta key ``download_fail_on_dataloss``.
 2017-03-01 11:43:53 -03:00
Paul Tremberth 2f28cb3b4a Merge pull request #2587 from MikeinRealLife/documentation_update 
[MRG+1] Document ftp_user and ftp_password meta keys
 2017-03-01 12:51:12 +01:00
Mikhail Korobov 0e5ed21397 Merge pull request #2599 from redapple/py3-ignores-cleanup-ftp 
Fix FTP downloader and re-enable FTP tests on Python 3
 2017-02-28 15:39:42 +05:00
Paul Tremberth 0b90c3b43c Re-enable FTP tests on Python 3 2017-02-27 17:42:00 +01:00
Mikhail Korobov c72ba07326 Merge pull request #2543 from scrapy/retry-stats 
[MRG+1] Retry stats
 2017-02-27 21:03:35 +05:00
Konstantin Lopuhin 9a5479cbf7 Merge pull request #2596 from arvindch/patch-1 
Use single quotes uniformly
 2017-02-26 14:15:05 +04:00
Arvind Chembarpu e85f0db128 Use single quotes uniformly 
The default spider template mixes single and double quotes.
 2017-02-25 16:48:17 +05:30
Mikhail Korobov d6a8288a48 Merge pull request #2595 from Lodour/master 
Add omitted "self" arguments
 2017-02-25 13:12:21 +05:00
mangogao 4274f0d4d4 Add omitted "self" arguments 
“self” argument is omitted in some methods.
 2017-02-25 15:44:20 +08:00
MikeinRealLife 441f25507e fixed typo 
removed duplicate line
 2017-02-22 21:23:27 -08:00
MikeinRealLife 96a570a93a fixed ticket #2574 2017-02-22 21:17:34 -08:00
Mikhail Korobov ce8a9aadc9 Merge pull request #2566 from scrapy/httperror-stats 
[MRG+1] HttpErrorMiddleware stats
 2017-02-22 18:55:42 +05:00