RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,450 Commits 49 Branches 132 Tags 160 MiB
Commit Graph

709 Commits

Author SHA1 Message Date
Konstantin Lopuhin 6f55ca4643 Revert unneeded test_crawl changes 2017-03-07 14:20:52 +03:00
Paul Tremberth 11cdf58abe Always decompress Content-Encoding: gzip at HttpCompression stage 
Let SitemapSpider handle decoding of .xml.gz files if necessary
 2017-03-07 11:02:46 +01:00
Paul Tremberth e42b846a9f Use body to chose response type after decompression content 2017-03-06 23:10:38 +01:00
Paul Tremberth 768f3155e5 Fix referrer policy from response headers and support explicit empty string 2017-03-06 16:20:37 +01:00
Paul Tremberth 0a17f9b55c Merge pull request #2334 from ArturGaspar/data_uri 
[MRG+1] data URI download handler.
 2017-03-06 13:50:46 +01:00
Paul Tremberth c68f99eed8 Refactor settings tests 2017-03-03 17:03:25 +01:00
Konstantin Lopuhin ef04cfd237 Respect log settings in custom_settings: fixes GH-1612 
A new root logger is installed when a crawler is created
if one was already installed before.
This allows to respect custom settings related to logging,
such as LOG_LEVEL, LOG_FILE, etc.
 2017-03-03 18:01:11 +03:00
Paul Tremberth f7e11b198e Cleanup 2017-03-03 16:00:59 +01:00
Paul Tremberth ecde166ee1 Refactor without MEDIA_HTTPSTATUS_LIST setting 2017-03-03 15:52:05 +01:00
Bernardas 11b31c9fbd fix redirect change 2017-03-03 15:52:05 +01:00
Bernardas 3cef1cd451 adjust variable wording and redirect logic 2017-03-03 15:52:05 +01:00
Bernardas 6a42214716 add tests for media pipeline MEDIA_ALLOW_REDIRECTS and MEDIA_HTTPSTATUS_LIST settings 2017-03-03 15:52:05 +01:00
Mikhail Korobov 7e8453cf1e Merge pull request #2306 from redapple/referrer-policy 
[MRG] Referrer policies in RefererMiddleware
 2017-03-03 04:05:13 +05:00
Mikhail Korobov 93024c242b Merge pull request #2537 from scrapy/no-canonicalize 
[MRG+1] Set canonicalize=False for LinkExtractor
 2017-03-03 02:53:36 +05:00
Artur Gaspar b50d0370f4 Test response attributes in data URI download handler. 2017-03-02 14:46:33 -03:00
Paul Tremberth 12a8ddecab Fix tests 2017-03-02 13:03:18 +01:00
Paul Tremberth e71803c833 Add tests for duplicate spider name warnings 2017-03-02 12:48:47 +01:00
Paul Tremberth 2d55d838ca Fix strip_url() tests 2017-03-01 20:59:52 +01:00
Paul Tremberth efa50039ec Add tests for policy fallback on unknown policies from meta and headers 2017-03-01 17:51:23 +01:00
Paul Tremberth 8226e77010 Add test for Referer header on HTTP redirections 2017-03-01 17:51:23 +01:00
Paul Tremberth d2aa51c0fb Update tests 2017-03-01 17:51:23 +01:00
Paul Tremberth bc200d1155 Rename setting to REFERRER_POLICY (with 2 Rs) 2017-03-01 17:51:23 +01:00
Paul Tremberth b6c761d2b4 Fix tests 2017-03-01 17:50:39 +01:00
Paul Tremberth 5cef67ae75 Update Referrer tests for "strict-" policies 2017-03-01 17:50:39 +01:00
Paul Tremberth 0a0b60a59f Add tests for stripping userinfo with percent-encoded delimiters 2017-03-01 17:50:39 +01:00
Paul Tremberth 8864d0e8c1 Rename helper function to strip_url() + add more tests 2017-03-01 17:50:39 +01:00
Paul Tremberth 5dd7311cd4 Move URL credentials stripping to a helper function 2017-03-01 17:50:39 +01:00
Paul Tremberth d3d4d66ce8 Add tests for referrer-policy set in response HTTP headers 2017-03-01 17:50:39 +01:00
Paul Tremberth e50e670eff Add test for custom referrer policy via settings 2017-03-01 17:50:39 +01:00
Paul Tremberth 0344f57fef Support case-insensitive policy names in settings 2017-03-01 17:50:39 +01:00
Paul Tremberth e72b6e3361 Add tests for referrer policy via settings and via Request meta 2017-03-01 17:50:39 +01:00
Paul Tremberth 7ec1b5f6c3 Add tests for the different referrer policies 2017-03-01 17:50:38 +01:00
Paul Tremberth baed7c436f WIP Add Referrer policies 2017-03-01 17:50:38 +01:00
Rolando Espinoza f01ae6ffcd Handle data loss gracefully. 
Websites that return a wrong ``Content-Length`` header may cause a data
loss error. Also when a chunked response is not finished properly.

This change adds a new setting ``DOWNLOAD_FAIL_ON_DATALOSS`` (default:
``True``) and request.meta key ``download_fail_on_dataloss``.
 2017-03-01 11:43:53 -03:00
Mikhail Korobov 0e5ed21397 Merge pull request #2599 from redapple/py3-ignores-cleanup-ftp 
Fix FTP downloader and re-enable FTP tests on Python 3
 2017-02-28 15:39:42 +05:00
Paul Tremberth 0b90c3b43c Re-enable FTP tests on Python 3 2017-02-27 17:42:00 +01:00
Mikhail Korobov c72ba07326 Merge pull request #2543 from scrapy/retry-stats 
[MRG+1] Retry stats
 2017-02-27 21:03:35 +05:00
Artur Gaspar 121a668a47 Rename data URI downloader module. 2017-02-22 12:30:56 -03:00
Artur Gaspar 3397d27574 Test for binary body content from data URI downloader. 2017-02-22 12:30:56 -03:00
Artur Gaspar d60642e175 data URI download handler. 2017-02-22 12:30:56 -03:00
Mikhail Korobov 26c1256f95 HttpErrorMiddleware stats 2017-02-22 17:36:25 +05:00
Mikhail Korobov 2b4d46315f TST fixed compatibility with new link extractor whitespace handling 2017-02-21 00:05:40 +05:00
Mikhail Korobov df446d167f fix deprecated link extractors 2017-02-20 23:01:14 +05:00
Mikhail Korobov 47f7da8724 canonicalize=False by default for LinkExtractor. Fixes GH-1941. 2017-02-20 22:58:11 +05:00
Mikhail Korobov 93e449f1f6 Merge pull request #2343 from redapple/anonymous-ftp 
[MRG+1] Support Anonymous FTP
 2017-02-20 23:19:54 +06:00
Omer Schleifer ff3e299eb0 [MRG+2] add flags to request (#2082) 
* add flags to request

* fxi test - add flags to request

* fix test(2) - add flags to request

* fix test(2) - add flags to request

* Updated test to reqser with flags field of request

* Updated documntation with flags field of request

* fix test identation

* fix test failed

* make the change backward comptaible

* remove  unrequired  spaces, fix documentation request flags

* remove  unrequired  space

* fx assert equal

* flags default is empty list

* Add flags to request

* add flags to request

* fxi test - add flags to request

* fix test(2) - add flags to request

* fix test(2) - add flags to request

* Updated test to reqser with flags field of request

* Updated documntation with flags field of request

* fix test identation

* fix test failed

* make the change backward comptaible

* remove  unrequired  spaces, fix documentation request flags

* remove  unrequired  space

* fx assert equal

* flags default is empty list

* add flags to request squashed commits
 2017-02-20 20:42:29 +06:00
Daniel Graña b0388e49b4 Merge pull request #1728 from scrapy/deprecate-make-requests-from-url 
deprecate Spider.make_requests_from_url.
 2017-02-20 11:23:49 -03:00
Daniel Graña c68140e68a Merge pull request #2540 from scrapy/response-follow 
response.follow
 2017-02-20 11:21:21 -03:00
Daniel Graña b15b4541d1 Merge pull request #2569 from scrapy/fix-project-util-test-osx 
TST fixed ProjectUtilsTest on OS X
 2017-02-20 11:15:21 -03:00
Daniel Graña 322fd68e4c Merge pull request #2548 from scrapy/formrequest-whitespaces 
[MRG+1] FormRequest: handle whitespaces in action attribute properly
 2017-02-20 11:09:50 -03:00
Daniel Graña 4a93be4ad8 Merge pull request #2547 from scrapy/linkextractor-strip-whitespaces 
[MRG+1] LinkExtractors: strip whitespaces
 2017-02-20 11:08:32 -03:00
Daniel Graña 58a18e3007 Merge pull request #2535 from pawelmhm/brotli 
[MRG+1] [httpcompression] add support for br - brotli content encoding
 2017-02-20 11:04:55 -03:00
Paul Tremberth f2e200120b Explicitly remove test directories in FTP tests 2017-02-20 14:50:05 +01:00
Paul Tremberth 26b7c039a9 Remove debug print statement 2017-02-20 14:19:36 +01:00
Paul Tremberth 3ffa2c5759 Support Anonymous FTP 
Fixes GH-2342
 2017-02-20 14:19:36 +01:00
Mikhail Korobov a1e8a8525d fix make_requests_from_url deprcation implementation, add tests 2017-02-17 00:18:29 +05:00
Mikhail Korobov 78cb46a079 TST fix a weird OS X testing issue 
pytest intercepts temp file creation to provide readable file/folder
names; path is built from method name; in case of conflicts
pytests uses increasing numbers, but it seems it doesn’t account
for case-insensitive (but case preserving) OS X filesystem. There
are methods named test_encoding, pytest thinks test_Encoding is
different and fails to create a test folder
 2017-02-16 18:22:04 +05:00
Mikhail Korobov b9928558b2 TST remove temporary files and folders 2017-02-16 17:50:38 +05:00
Mikhail Korobov f0c4e5dfd5 TST fixed ProjectUtilsTest on OS X 
Temp folder can be a symlink on OS X.
 2017-02-16 16:32:23 +05:00
Mikhail Korobov 692975acb4 deprecate Spider.make_requests_from_url. Fixes #1495. 2017-02-16 03:39:34 +05:00
Mikhail Korobov d079e15fe2 Strip leading/trailing whitespaces in link extractors. Fixes GH-838. 2017-02-16 02:22:17 +05:00
Mikhail Korobov 074caf434e FormRequest: handle whitespaces in action attribute properly 2017-02-16 02:13:39 +05:00
Mikhail Korobov fade5763af TST more response.follow tests 2017-02-16 02:02:50 +05:00
Mikhail Korobov 160da6abab fixed tests in Python 2 2017-02-15 04:41:53 +05:00
Mikhail Korobov 2674f317df Response.follow 2017-02-15 04:39:47 +05:00
Mikhail Korobov 608c3f0c45 handle whitespace in response.follow; add tests 2017-02-15 04:17:41 +05:00
Mikhail Korobov e285b1d6c2 retry stats 2017-02-14 20:27:49 +05:00
Mikhail Korobov afac3fd2c2 Merge pull request #2530 from elacuesta/proxy_credentials 
[MRG+1] Use credentials from request.meta['proxy']
 2017-02-14 20:58:39 +06:00
Eugenio Lacuesta ae0ea31abd Add HTTPPROXY_ENABLED setting (default True) 2017-02-14 11:33:01 -03:00
Paul Tremberth f32a229e3d Merge pull request #2510 from elacuesta/reqser_request_class 
[MRG+1] Preserve request class when converting to/from dicts (utils.reqser)
 2017-02-08 18:30:07 +01:00
Eugenio Lacuesta 9c0aae724e Use credentials from request.meta['proxy'] if present 2017-02-08 13:22:30 -03:00
Paul Tremberth a8893190a1 Merge pull request #2464 from elacuesta/component_order_integer 
[MRG+2] Validate numeric values (components order)
 2017-02-08 13:01:02 +01:00
Paul Tremberth 3becb861af Merge pull request #2544 from scrapy/twisted-version-cleanup 
[MRG+1] Remove code required to support ancient twisted versions.
 2017-02-08 12:59:45 +01:00
Rolando Espinoza 4e765acaed BUG: Fix __classcell__ propagation. 
Python 3.6 added simpler customization of class creation but this
requires to propagate correctly the __classcell__ attribute in custom
__new__ methods.

See https://docs.python.org/3.6/whatsnew/3.6.html#pep-487-simpler-
customization-of-class-creation
 2017-02-08 02:12:52 -04:00
Mikhail Korobov 04b2f79e7a Remove code required to support ancient twisted versions. See GH-1887. 2017-02-07 22:30:58 +05:00
Eugenio Lacuesta 24e82bfe75 Validate values for components order 2017-02-07 09:13:57 -03:00
Pawel Miech fb4ef21a1d [httpcompression] minor style edits 2017-02-07 10:22:42 +01:00
Pawel Miech af802bad14 [httpcompression] add brotlipy for python 3 2017-02-06 15:45:21 +01:00
Pawel Miech 3daf473686 [httpcompression] skip test if no brotli 2017-02-06 13:49:41 +01:00
Pawel Miech f8f8bbe080 [httpcompression] import brotli when available 2017-02-06 12:19:00 +01:00
Pawel Miech 16c4b4e184 [httpcompression] add support for br - brotli content encoding 2017-02-06 11:41:08 +01:00
Mikhail Korobov 814ce37dff Merge pull request #2528 from redapple/dns-retry-test 
Make DNS retry test compatible with Twisted 17+
 2017-02-03 02:43:46 +05:00
Paul Tremberth 02e1d2b1fd Add trailing dot 2017-02-02 22:28:37 +01:00
Paul Tremberth e604c0f3ab Remove unused imports 2017-02-02 18:26:41 +01:00
Paul Tremberth 3358254c5c Make DNS retry test compatible with Twisted 17+ 2017-02-02 17:53:28 +01:00
Paul Tremberth 803d8c4b57 Add tests for passing kwargs on response .xpath() shortcut 2017-02-02 17:44:57 +01:00
Paul Tremberth ae6d8d728e Support 'True' and 'False' strings as boolean settings values 2017-01-30 16:33:08 +01:00
Eugenio Lacuesta 53757e51e5 Preserve request class when converting to/from dicts (utils.reqser) 2017-01-24 17:13:21 -03:00
Paul Tremberth 07f9985a94 TST: Randomize FILES_EXPIRES above 90 days 2016-12-21 17:03:11 +01:00
Elias Dorneles d09ec3db68 Merge pull request #2410 from redapple/fetch-transparent-redirect 
[MRG+1] Transparently handle redirections in fetch and shell
 2016-12-21 09:49:15 -02:00
Mikhail Korobov d19c4c1f80 Merge pull request #2433 from redapple/wrong-spidermodules-warning 
[MRG+1] Warn user instead of failing for wrong SPIDER_MODULES setting
 2016-12-19 21:46:56 +05:00
Paul Tremberth ed1e4d8df3 Merge pull request #1731 from scrapy/disable-toplevel-2 
[MRG+1] LOG_SHORT_NAMES option to disable TopLevelFormatter
 2016-12-19 16:02:55 +01:00
Elias Dorneles 97e82107b1 Merge pull request #2270 from redapple/httperror-log-info 
[MRG+1] Raise log level for HttpErrorMiddleware to INFO (from DEBUG)
 2016-12-14 12:18:04 -02:00
Paul Tremberth f7e4081414 Add tests for SequenceExclude container 2016-12-12 22:37:53 +01:00
Mikhail Korobov 05b4555f39 TST tests for LOG_SHORT_NAMES 2016-12-09 02:19:51 +05:00
Mikhail Korobov e46572d6f2 TST end-to-end test for LOG_LEVEL option 
there were no end-to-end tests for this option
 2016-12-09 02:19:33 +05:00
Mikhail Korobov 6eab59cbac TST cleanup runspider tests 2016-12-09 02:14:12 +05:00
Paul Tremberth 948e3cd003 Warn user instead of failing for wrong SPIDER_MODULES setting 2016-12-08 12:50:26 +01:00
Paul Tremberth 2cd579a774 Add test for fetch(url) within shell with and without redirect 2016-12-07 19:07:32 +01:00
Paul Tremberth 7e54de2455 Add tests for shell command with and without --no-redirect 2016-12-07 18:41:24 +01:00