RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,172 Commits 50 Branches 132 Tags 142 MiB
Commit Graph

6172 Commits

Author SHA1 Message Date
Paul Tremberth f2ee6be3bb Use urlparse_cached() for OriginPolicy 2017-03-01 17:50:39 +01:00
Paul Tremberth 3af88a2877 Use urlparse_cached() on request and responses 2017-03-01 17:50:39 +01:00
Paul Tremberth 7ec1b5f6c3 Add tests for the different referrer policies 2017-03-01 17:50:38 +01:00
Paul Tremberth baed7c436f WIP Add Referrer policies 2017-03-01 17:50:38 +01:00
Mikhail Korobov 7b49b9c0f5 Merge pull request #2590 from rolando-contrib/handle-data-loss-gracefully 
[MRG+2] Handle data loss gracefully.
 2017-03-01 20:23:19 +05:00
Rolando Espinoza f01ae6ffcd Handle data loss gracefully. 
Websites that return a wrong ``Content-Length`` header may cause a data
loss error. Also when a chunked response is not finished properly.

This change adds a new setting ``DOWNLOAD_FAIL_ON_DATALOSS`` (default:
``True``) and request.meta key ``download_fail_on_dataloss``.
 2017-03-01 11:43:53 -03:00
Paul Tremberth 2f28cb3b4a Merge pull request #2587 from MikeinRealLife/documentation_update 
[MRG+1] Document ftp_user and ftp_password meta keys
 2017-03-01 12:51:12 +01:00
Mikhail Korobov 0e5ed21397 Merge pull request #2599 from redapple/py3-ignores-cleanup-ftp 
Fix FTP downloader and re-enable FTP tests on Python 3
 2017-02-28 15:39:42 +05:00
Paul Tremberth 0b90c3b43c Re-enable FTP tests on Python 3 2017-02-27 17:42:00 +01:00
Mikhail Korobov c72ba07326 Merge pull request #2543 from scrapy/retry-stats 
[MRG+1] Retry stats
 2017-02-27 21:03:35 +05:00
Konstantin Lopuhin 9a5479cbf7 Merge pull request #2596 from arvindch/patch-1 
Use single quotes uniformly
 2017-02-26 14:15:05 +04:00
Arvind Chembarpu e85f0db128 Use single quotes uniformly 
The default spider template mixes single and double quotes.
 2017-02-25 16:48:17 +05:30
Mikhail Korobov d6a8288a48 Merge pull request #2595 from Lodour/master 
Add omitted "self" arguments
 2017-02-25 13:12:21 +05:00
mangogao 4274f0d4d4 Add omitted "self" arguments 
“self” argument is omitted in some methods.
 2017-02-25 15:44:20 +08:00
MikeinRealLife 441f25507e fixed typo 
removed duplicate line
 2017-02-22 21:23:27 -08:00
MikeinRealLife 96a570a93a fixed ticket #2574 2017-02-22 21:17:34 -08:00
Mikhail Korobov ce8a9aadc9 Merge pull request #2566 from scrapy/httperror-stats 
[MRG+1] HttpErrorMiddleware stats
 2017-02-22 18:55:42 +05:00
Mikhail Korobov 26c1256f95 HttpErrorMiddleware stats 2017-02-22 17:36:25 +05:00
Mikhail Korobov cac51190df Merge pull request #2577 from redapple/cleanup-ctxfactory 
Set context factory implementation based on Twisted version
 2017-02-21 13:51:36 +05:00
Mikhail Korobov 565d1ca766 Merge pull request #2495 from redapple/proxy-connect-headers-buffer 
[MRG] Buffer CONNECT response bytes from proxy until all HTTP headers are received
 2017-02-21 02:44:36 +06:00
Mikhail Korobov 93e449f1f6 Merge pull request #2343 from redapple/anonymous-ftp 
[MRG+1] Support Anonymous FTP
 2017-02-20 23:19:54 +06:00
Paul Tremberth f3a7567443 Add note on FTP_PASSWORD default value 2017-02-20 17:15:05 +01:00
Paul Tremberth 301847d862 Set context factory implementation based on Twisted version 2017-02-20 16:42:57 +01:00
Paul Tremberth adb180fbaf Use bytearray for the CONNECT message bytes buffer 2017-02-20 16:25:12 +01:00
Paul Tremberth 1d80efde89 Merge pull request #2159 from scrapy/remove-prerelease-configuration 
[MRG+1] Remove bumpversion prerelease configuration
 2017-02-20 16:15:05 +01:00
Omer Schleifer ff3e299eb0 [MRG+2] add flags to request (#2082) 
* add flags to request

* fxi test - add flags to request

* fix test(2) - add flags to request

* fix test(2) - add flags to request

* Updated test to reqser with flags field of request

* Updated documntation with flags field of request

* fix test identation

* fix test failed

* make the change backward comptaible

* remove  unrequired  spaces, fix documentation request flags

* remove  unrequired  space

* fx assert equal

* flags default is empty list

* Add flags to request

* add flags to request

* fxi test - add flags to request

* fix test(2) - add flags to request

* fix test(2) - add flags to request

* Updated test to reqser with flags field of request

* Updated documntation with flags field of request

* fix test identation

* fix test failed

* make the change backward comptaible

* remove  unrequired  spaces, fix documentation request flags

* remove  unrequired  space

* fx assert equal

* flags default is empty list

* add flags to request squashed commits
 2017-02-20 20:42:29 +06:00
Daniel Graña b0388e49b4 Merge pull request #1728 from scrapy/deprecate-make-requests-from-url 
deprecate Spider.make_requests_from_url.
 2017-02-20 11:23:49 -03:00
Daniel Graña c68140e68a Merge pull request #2540 from scrapy/response-follow 
response.follow
 2017-02-20 11:21:21 -03:00
Daniel Graña 377db31a32 Merge pull request #2557 from scrapy/gitignore 
add a couple more lines to gitignore
 2017-02-20 11:18:48 -03:00
Daniel Graña e578be73b6 Merge pull request #2539 from scrapy/enable-memusage 
Enable memusage extension by default.
 2017-02-20 11:18:04 -03:00
Paul Tremberth 4a1b88fe13 Merge pull request #2567 from scrapy/no-py3-badge 
remove “Python 3 progress” badge
 2017-02-20 15:16:40 +01:00
Daniel Graña b15b4541d1 Merge pull request #2569 from scrapy/fix-project-util-test-osx 
TST fixed ProjectUtilsTest on OS X
 2017-02-20 11:15:21 -03:00
Daniel Graña fab168bbfb Merge pull request #2572 from advarisk/warning-formrequest-from-response 
[MRG+1] document issue with FormRequest.from_response due to bug in lxml
 2017-02-20 11:14:28 -03:00
Daniel Graña 85ef6a6229 Merge pull request #2564 from elacuesta/docs_exporters 
[MRG+1] Doc: binary mode is required for exporters
 2017-02-20 11:12:48 -03:00
Daniel Graña b2d66dc1f0 Merge pull request #2562 from terut/feature/build_request 
[MRG+1] Separate building request from _requests_to_follow in CrawlSpider
 2017-02-20 11:12:10 -03:00
Daniel Graña 322fd68e4c Merge pull request #2548 from scrapy/formrequest-whitespaces 
[MRG+1] FormRequest: handle whitespaces in action attribute properly
 2017-02-20 11:09:50 -03:00
Daniel Graña 4a93be4ad8 Merge pull request #2547 from scrapy/linkextractor-strip-whitespaces 
[MRG+1] LinkExtractors: strip whitespaces
 2017-02-20 11:08:32 -03:00
Daniel Graña 58a18e3007 Merge pull request #2535 from pawelmhm/brotli 
[MRG+1] [httpcompression] add support for br - brotli content encoding
 2017-02-20 11:04:55 -03:00
Mikhail Korobov 7ad5156287 Merge pull request #2570 from scrapy/tests-cleanup-resources 
[MRG+1] TST remove temp files and folders
 2017-02-20 20:00:32 +06:00
Paul Tremberth f2e200120b Explicitly remove test directories in FTP tests 2017-02-20 14:50:05 +01:00
Paul Tremberth d35a01a103 Update default password 2017-02-20 14:23:23 +01:00
Paul Tremberth b80e1bb6c5 Document new FTP_* settings 2017-02-20 14:19:36 +01:00
Paul Tremberth 565baae1e2 Change FTP_* settings names 2017-02-20 14:19:36 +01:00
Paul Tremberth 26b7c039a9 Remove debug print statement 2017-02-20 14:19:36 +01:00
Paul Tremberth 3ffa2c5759 Support Anonymous FTP 
Fixes GH-2342
 2017-02-20 14:19:36 +01:00
Ashish Kulkarni 165e2cb8c9 document issue with FormRequest.from_response due to bug in lxml 
This can make the spider fail due to incorrect values being posted
server-side, which is extremely hard to debug because it is easy
to miss leading/trailing whitespace, even with a logging proxy.

The fix was merged for lxml 3.8 in lxml/lxml#228 so document that
as well.
 2017-02-17 14:54:22 +05:30
Mikhail Korobov 5adacc4dde fixed method override check in Python 2 2017-02-17 02:35:02 +05:00
Mikhail Korobov a1e8a8525d fix make_requests_from_url deprcation implementation, add tests 2017-02-17 00:18:29 +05:00
Mikhail Korobov 78cb46a079 TST fix a weird OS X testing issue 
pytest intercepts temp file creation to provide readable file/folder
names; path is built from method name; in case of conflicts
pytests uses increasing numbers, but it seems it doesn’t account
for case-insensitive (but case preserving) OS X filesystem. There
are methods named test_encoding, pytest thinks test_Encoding is
different and fails to create a test folder
 2017-02-16 18:22:04 +05:00
Mikhail Korobov b9928558b2 TST remove temporary files and folders 2017-02-16 17:50:38 +05:00