RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,396 Commits 49 Branches 132 Tags 187 MiB
Commit Graph

6396 Commits

Author SHA1 Message Date
Paul Tremberth 5dd7311cd4 Move URL credentials stripping to a helper function 2017-03-01 17:50:39 +01:00
Paul Tremberth c9c59db489 Update documentation about REFERER_POLICY setting 2017-03-01 17:50:39 +01:00
Paul Tremberth 285d5bc03a Patch "Referer" header on HTTP redirects if necessary 2017-03-01 17:50:39 +01:00
Paul Tremberth d3d4d66ce8 Add tests for referrer-policy set in response HTTP headers 2017-03-01 17:50:39 +01:00
Paul Tremberth e50e670eff Add test for custom referrer policy via settings 2017-03-01 17:50:39 +01:00
Paul Tremberth ec8b4c1a9b Change __init__ default "settings" arg handling 2017-03-01 17:50:39 +01:00
Paul Tremberth 0344f57fef Support case-insensitive policy names in settings 2017-03-01 17:50:39 +01:00
Paul Tremberth e72b6e3361 Add tests for referrer policy via settings and via Request meta 2017-03-01 17:50:39 +01:00
Paul Tremberth 842ce131aa Make default referrer policy customizable via settings 2017-03-01 17:50:39 +01:00
Paul Tremberth f6205778f3 Refactor ReferrerPolicy methods 2017-03-01 17:50:39 +01:00
Paul Tremberth f6a800fde6 Remove all non-cached urlparsing references 2017-03-01 17:50:39 +01:00
Paul Tremberth 59cb884ace Use urlparse_cached() for OriginWhenCrossOriginPolicy 2017-03-01 17:50:39 +01:00
Paul Tremberth f2ee6be3bb Use urlparse_cached() for OriginPolicy 2017-03-01 17:50:39 +01:00
Paul Tremberth 3af88a2877 Use urlparse_cached() on request and responses 2017-03-01 17:50:39 +01:00
Paul Tremberth 7ec1b5f6c3 Add tests for the different referrer policies 2017-03-01 17:50:38 +01:00
Paul Tremberth baed7c436f WIP Add Referrer policies 2017-03-01 17:50:38 +01:00
Mikhail Korobov 7b49b9c0f5 Merge pull request #2590 from rolando-contrib/handle-data-loss-gracefully 
[MRG+2] Handle data loss gracefully.
 2017-03-01 20:23:19 +05:00
Rolando Espinoza f01ae6ffcd Handle data loss gracefully. 
Websites that return a wrong ``Content-Length`` header may cause a data
loss error. Also when a chunked response is not finished properly.

This change adds a new setting ``DOWNLOAD_FAIL_ON_DATALOSS`` (default:
``True``) and request.meta key ``download_fail_on_dataloss``.
 2017-03-01 11:43:53 -03:00
Paul Tremberth 2f28cb3b4a Merge pull request #2587 from MikeinRealLife/documentation_update 
[MRG+1] Document ftp_user and ftp_password meta keys
 2017-03-01 12:51:12 +01:00
Mikhail Korobov 0e5ed21397 Merge pull request #2599 from redapple/py3-ignores-cleanup-ftp 
Fix FTP downloader and re-enable FTP tests on Python 3
 2017-02-28 15:39:42 +05:00
Paul Tremberth 0b90c3b43c Re-enable FTP tests on Python 3 2017-02-27 17:42:00 +01:00
Mikhail Korobov c72ba07326 Merge pull request #2543 from scrapy/retry-stats 
[MRG+1] Retry stats
 2017-02-27 21:03:35 +05:00
Konstantin Lopuhin 9a5479cbf7 Merge pull request #2596 from arvindch/patch-1 
Use single quotes uniformly
 2017-02-26 14:15:05 +04:00
Arvind Chembarpu e85f0db128 Use single quotes uniformly 
The default spider template mixes single and double quotes.
 2017-02-25 16:48:17 +05:30
Mikhail Korobov d6a8288a48 Merge pull request #2595 from Lodour/master 
Add omitted "self" arguments
 2017-02-25 13:12:21 +05:00
mangogao 4274f0d4d4 Add omitted "self" arguments 
“self” argument is omitted in some methods.
 2017-02-25 15:44:20 +08:00
MikeinRealLife 441f25507e fixed typo 
removed duplicate line
 2017-02-22 21:23:27 -08:00
MikeinRealLife 96a570a93a fixed ticket #2574 2017-02-22 21:17:34 -08:00
Artur Gaspar 3139f4a5f7 Add data URI download handler to settings. 2017-02-22 12:30:56 -03:00
Artur Gaspar 121a668a47 Rename data URI downloader module. 2017-02-22 12:30:56 -03:00
Artur Gaspar c847e7d4d0 Use w3lib data URI parser. 2017-02-22 12:30:56 -03:00
Artur Gaspar 3397d27574 Test for binary body content from data URI downloader. 2017-02-22 12:30:56 -03:00
Artur Gaspar 7e9f2c31d7 Ensure bytes objects when needed in data URI downloader. 2017-02-22 12:30:56 -03:00
Artur Gaspar d60642e175 data URI download handler. 2017-02-22 12:30:56 -03:00
Mikhail Korobov ce8a9aadc9 Merge pull request #2566 from scrapy/httperror-stats 
[MRG+1] HttpErrorMiddleware stats
 2017-02-22 18:55:42 +05:00
Mikhail Korobov 26c1256f95 HttpErrorMiddleware stats 2017-02-22 17:36:25 +05:00
Mikhail Korobov cac51190df Merge pull request #2577 from redapple/cleanup-ctxfactory 
Set context factory implementation based on Twisted version
 2017-02-21 13:51:36 +05:00
Mikhail Korobov 565d1ca766 Merge pull request #2495 from redapple/proxy-connect-headers-buffer 
[MRG] Buffer CONNECT response bytes from proxy until all HTTP headers are received
 2017-02-21 02:44:36 +06:00
Mikhail Korobov 2b4d46315f TST fixed compatibility with new link extractor whitespace handling 2017-02-21 00:05:40 +05:00
Mikhail Korobov df446d167f fix deprecated link extractors 2017-02-20 23:01:14 +05:00
Mikhail Korobov 47f7da8724 canonicalize=False by default for LinkExtractor. Fixes GH-1941. 2017-02-20 22:58:11 +05:00
Mikhail Korobov bb7d99ed81 drop unneeded urlparse call 2017-02-20 22:54:14 +05:00
Mikhail Korobov 93e449f1f6 Merge pull request #2343 from redapple/anonymous-ftp 
[MRG+1] Support Anonymous FTP
 2017-02-20 23:19:54 +06:00
Paul Tremberth f3a7567443 Add note on FTP_PASSWORD default value 2017-02-20 17:15:05 +01:00
Paul Tremberth 301847d862 Set context factory implementation based on Twisted version 2017-02-20 16:42:57 +01:00
Paul Tremberth adb180fbaf Use bytearray for the CONNECT message bytes buffer 2017-02-20 16:25:12 +01:00
Paul Tremberth 1d80efde89 Merge pull request #2159 from scrapy/remove-prerelease-configuration 
[MRG+1] Remove bumpversion prerelease configuration
 2017-02-20 16:15:05 +01:00
Omer Schleifer ff3e299eb0 [MRG+2] add flags to request (#2082) 
* add flags to request

* fxi test - add flags to request

* fix test(2) - add flags to request

* fix test(2) - add flags to request

* Updated test to reqser with flags field of request

* Updated documntation with flags field of request

* fix test identation

* fix test failed

* make the change backward comptaible

* remove  unrequired  spaces, fix documentation request flags

* remove  unrequired  space

* fx assert equal

* flags default is empty list

* Add flags to request

* add flags to request

* fxi test - add flags to request

* fix test(2) - add flags to request

* fix test(2) - add flags to request

* Updated test to reqser with flags field of request

* Updated documntation with flags field of request

* fix test identation

* fix test failed

* make the change backward comptaible

* remove  unrequired  spaces, fix documentation request flags

* remove  unrequired  space

* fx assert equal

* flags default is empty list

* add flags to request squashed commits
 2017-02-20 20:42:29 +06:00
Daniel Graña b0388e49b4 Merge pull request #1728 from scrapy/deprecate-make-requests-from-url 
deprecate Spider.make_requests_from_url.
 2017-02-20 11:23:49 -03:00
Daniel Graña c68140e68a Merge pull request #2540 from scrapy/response-follow 
response.follow
 2017-02-20 11:21:21 -03:00