RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,614 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

479 Commits

Author SHA1 Message Date
Paul Tremberth d843a0aae8 Amend "settings" command to output JSON for dict settings 2016-01-29 21:12:03 +01:00
Paul Tremberth aa78758bc7 Update tests for settings copy_to_dict() 2016-01-29 18:59:12 +01:00
Mikhail Korobov a35aec71e9 Merge pull request #1710 from redapple/1550-shell_file-cont 
[MRG+1] shell command's ability to open local files + tests
 2016-01-29 21:03:40 +05:00
Paul Tremberth 78f00401cd Remove unused import in tests 2016-01-29 16:56:05 +01:00
stummjr cf2ebb0687 Include tests for exporters: JSON, JSON-Lines, Pickle and Marshal. 2016-01-28 19:43:53 -02:00
Paul Tremberth c6f374f2eb Merge remote-tracking branch 'origin/master' into 1550-shell_file-cont 2016-01-28 14:02:48 +01:00
Paul Tremberth 481e251775 Move guess_scheme() tests to relevant test module 2016-01-28 13:51:50 +01:00
Paul Tremberth cae268402d Move guess_scheme() to scrapy.utils.url 2016-01-28 13:42:04 +01:00
stummjr c55ff110a3 Fix CSV exporter for non string Python types. 2016-01-27 15:43:17 -02:00
stummjr f1d971a5c0 fix PythonItemExporter for non-string types 2016-01-27 14:34:46 -02:00
Artur Gaspar dc8701ea42 Add test for already failed deferreds when downloading page in robots.txt middleware. 2016-01-27 12:56:42 -02:00
Mikhail Korobov 4bcbb77bcc response.text. Fixes GH-1729. 2016-01-27 01:28:11 +05:00
Paul Tremberth 29695375d1 Add test for raised exception with SpiderState extension when no JOBDIR used 2016-01-26 16:33:24 +01:00
Νικόλαος-Διγενής Καραγιάννης 1cffa99e0d tests+doc for subdomains in offsite middleware 2016-01-26 12:49:43 +02:00
Daniel Graña a7b86137d0 Merge pull request #1499 from scrapy/py3-port-exporters 
[MRG+1] PY3 exporters
 2016-01-26 01:01:30 -03:00
Daniel Graña d0eacfe0f9 Add test case for marshal item exporter 2016-01-26 00:26:27 -03:00
Elias Dorneles 2dfdde3c79 fallback to repr when can't convert to native string 2016-01-25 22:24:35 -02:00
Elias Dorneles 23b3336c1f add test for invalid option 2016-01-25 22:11:04 -02:00
Daniel Graña b20463183c Merge pull request #1716 from lopuhin/py3-test-pipeline-files-images 
[MRG+1] py3: test_pipeline_files and test_pipeline_images
 2016-01-25 18:24:12 -03:00
Elias Dorneles 0c44fac2b5 added tests for feed export marshal and pickle 2016-01-24 19:17:42 -02:00
Konstantin Lopuhin 1be90323c2 py3: properly skip s3 tests on py3 2016-01-24 23:44:56 +03:00
Konstantin Lopuhin 4233b3cda4 py3: reviewed passing test_spidermiddleware_httperror.py 2016-01-24 23:10:03 +03:00
Konstantin Lopuhin 097082cffa reviewed py3 compat in pipelines/images.py and pipelines/files.py 2016-01-24 23:05:23 +03:00
Konstantin Lopuhin 333d4c91fb py3: add boto to py3 test requirements, test_pipeline_files and test_pipeline_images passing now 2016-01-24 22:55:21 +03:00
Konstantin Lopuhin 860353b0c0 py3: unskip test_mail and scrapy/mail.py 2016-01-24 13:27:41 +03:00
Konstantin Lopuhin 9704226ee4 py3: fix test_mail - get_payload returns bytes when decode is True 2016-01-24 13:25:14 +03:00
Elias Dorneles 9fbe6f3e81 added feedexport test for xml output 2016-01-23 17:17:40 -02:00
Elias Dorneles c75f1fe46a restore bytes instead of text, for easier reviewing 2016-01-23 16:09:57 -02:00
Paul Tremberth 5f09da60c1 Revert "Use pytest.mark.parametrize decorator" 
This reverts commit 1a30a7774b.
 2016-01-22 23:48:58 +01:00
Paul Tremberth 1a30a7774b Use pytest.mark.parametrize decorator 2016-01-22 18:22:19 +01:00
Paul Tremberth 61ca8d587a Merge remote-tracking branch 'origin/master' into 1550-shell_file-cont 2016-01-22 18:16:36 +01:00
Paul Tremberth 7a51d370f3 Regex-based guess_scheme() + refactor tests 2016-01-22 17:16:27 +01:00
Elias Dorneles d0955fd083 add back test for latin-1 encoding 2016-01-22 10:07:55 -02:00
Paul Tremberth 6d73e057b5 Extract guess_scheme function and refactor tests 2016-01-22 13:07:42 +01:00
Mikhail Korobov 35ada10729 PY3 enable tests for scrapy parse command 
scrapy parse command is already ported
 2016-01-22 13:39:27 +05:00
Elias Dorneles e938752973 add test for PythonItemExporter binary mode 2016-01-21 21:51:59 -02:00
Elias Dorneles 2514973242 re-enable skipped feed export tests 2016-01-21 21:22:12 -02:00
Elias Dorneles b746d85f4c PY3 port csv exporter 2016-01-21 21:12:43 -02:00
Paul Tremberth 8bd5b60889 Remove relpath filepath 2016-01-21 23:23:50 +01:00
Paul Tremberth 240ecbf323 Add local file tests for scrapy shell command 
Continuation of #1579
 2016-01-21 22:59:48 +01:00
Elias Dorneles c76190d491 PY3: ported json(lines), xml exporters 2016-01-21 18:24:06 -02:00
Elias Dorneles b6ef1f19fd make BaseItemExporter export unicode, pushed down previous behavior for classes that need it 2016-01-21 15:36:14 -02:00
Elias Dorneles a76ecd4ef0 remove test_exporters from py3 ignores 2016-01-21 15:36:14 -02:00
Konstantin Lopuhin 5813de8838 py3: unskip test_closespider - it passes after fixing mockserver.Follow resouce on py3 2016-01-21 17:53:08 +03:00
Konstantin Lopuhin 4607f2843e py3: unskip test_crawl 2016-01-21 17:53:08 +03:00
Konstantin Lopuhin bf5f54fa33 py3: fix getarg 2016-01-21 17:52:12 +03:00
Konstantin Lopuhin ad2b3321b9 py3 compat: use range, fixes CrawlTestCase.test_start_requests_bug_yielding 2016-01-21 17:52:12 +03:00
Konstantin Lopuhin 0680950b98 py3: pass CrawlTestCase.test_referer_header, fixing Echo resource in mockserver and json decoding in test 2016-01-21 17:52:12 +03:00
Konstantin Lopuhin 20b839b44b py3: pass first crawl test (test_follow_all): fix mock server 2016-01-21 17:52:12 +03:00
Daniel Graña 0f500a1f82 Merge pull request #1708 from scrapy/fix-scrapy-bench 
PY3 fixed scrapy bench command
 2016-01-21 11:51:27 -03:00
Raul Gallegos a06a5f00f4 adding configurable encoding for httpproxy authentication 2016-01-21 09:35:04 -05:00
Raul Gallegos f042ad0f39 py3 fix HttpProxy and Retry Middlewares 2016-01-21 09:35:04 -05:00
Paul Tremberth c9e046d11d Merge pull request #1657 from palego/os.mknod-fails-on-macos 
[MRG+1] change os.mknod() for open()
 2016-01-21 15:17:57 +01:00
Mikhail Korobov 80c55f19a1 PY3 fixed scrapy bench command 2016-01-21 18:31:58 +05:00
Paul Tremberth da83f2dca7 Merge pull request #1693 from lopuhin/py3-commands-fetch-shell 
[MRG+1] Py3: port commands fetch and shell
 2016-01-21 10:59:32 +01:00
Paul Tremberth 6d8620ffe7 Merge pull request #1691 from lopuhin/py3-test-engine 
[MRG+1] Py3: port test_engine
 2016-01-21 10:30:39 +01:00
Capi Etheriel 659715ecd9 implements FormRequest.from_response CSS support 2016-01-21 01:05:20 -02:00
Konstantin Lopuhin a5da7531c4 py3 backout skipping test_closespider - it was fixed on another branch 2016-01-20 23:01:29 +03:00
Konstantin Lopuhin 47d3c63338 py3: port fetch and shell commands, and review + enable already passing test_closespider.py and tests/test_utils_template.py 2016-01-20 23:01:29 +03:00
Konstantin Lopuhin c1db60188a py3: unskip passing test_engine 2016-01-20 22:57:46 +03:00
Konstantin Lopuhin 0b08b4bfcf fix engine tests - this callback (spider_closed_callback) should accept one argument, but the error was hidden on py2 2016-01-20 22:57:46 +03:00
Konstantin Lopuhin a32b59ac75 py3: fix EngineTest.test_crawler 2016-01-20 22:57:46 +03:00
Elias Dorneles 8a1255d3ba Merge pull request #1680 from lopuhin/py3-downloader-middleware 
[MRG+1] Py3: port downloader cache and compression middlewares
 2016-01-20 17:05:54 -02:00
Daniel Graña 176610f910 optional_features has been removed 2016-01-19 15:34:26 -03:00
Daniel Graña 5ec4319885 Merge pull request #1358 from nyov/nyov/boto-anon-connection 
[MRG+1] Support anonymous S3DownloadHandler (boto) connections
 2016-01-19 13:35:23 -03:00
Konstantin Lopuhin 4c84172840 move leveldb to tests/requirements-py3.txt 2016-01-19 18:24:58 +03:00
Konstantin Lopuhin 07577c432e Merge branch 'master' into py3-http-downloaders 2016-01-19 17:39:22 +03:00
Daniel Graña 49fe631d89 Merge pull request #1676 from lopuhin/py3-webclient 
[MRG+1] py3: fix webclient
 2016-01-19 11:07:54 -03:00
Konstantin Lopuhin de98d8d006 move comment about test skipped on py3 into a proper place 2016-01-18 19:27:31 +03:00
Konstantin Lopuhin 7ffcc58759 Merge branch 'py3-http-downloaders' into py3-downloader-middleware 2016-01-18 19:23:12 +03:00
Konstantin Lopuhin 7fdd3225b2 fix test skipping logic - this is (temporary) py2-only part 2016-01-18 19:09:09 +03:00
Konstantin Lopuhin 324f2c1a6f common test_downloadermiddleware.py also passes now due to fixes in compression middleware 2016-01-18 16:45:22 +03:00
Konstantin Lopuhin 0b9336418e py3: port compression downloader middleware and tests 2016-01-18 16:43:58 +03:00
Konstantin Lopuhin 6edd4dec33 Merge branch 'py3-http-downloaders' into py3-downloader-middleware 2016-01-18 15:51:26 +03:00
Konstantin Lopuhin f876654a0c Merge branch 'py3-webclient' into py3-http-downloaders 2016-01-18 15:50:58 +03:00
Konstantin Lopuhin 4946434582 revert most changes to this test, and clarify - it is valid only on py2, because urls are strictly unicode on py3 2016-01-18 15:23:01 +03:00
Konstantin Lopuhin a2efd389b0 clarify: rename r_transform to response_transform 2016-01-18 15:09:54 +03:00
Konstantin Lopuhin b940606b7e this is a test for TunnelingTCP4ClientEndpoint - move into Http11ProxyTestCase 2016-01-18 15:06:15 +03:00
Konstantin Lopuhin 0f527849f2 https proxy tunneling - add a test (not perfect, but covers all impl) and fix for py3 2016-01-18 14:44:04 +03:00
Konstantin Lopuhin 98c060d0b2 py3: fix http 1.1 test with https, and use self.host everywhere 2016-01-18 12:42:21 +03:00
Konstantin Lopuhin 04f69fd184 add https 1.1 downloader test - localhost is a valid DNS-ID 2016-01-18 12:37:46 +03:00
Konstantin Lopuhin f403c88053 Merge branch 'py3-webclient' into py3-http-downloaders 2016-01-18 12:25:51 +03:00
palego 6b905a9aec split-up the assertIn 
to deal with OS X intricacies (directories prefixed with /private)
 2016-01-18 10:18:31 +01:00
Konstantin Lopuhin 673df5e416 add webclient test - check that non-standart body encoding matches Content-Encoding header 2016-01-18 11:44:49 +03:00
Konstantin Lopuhin 7d44c5dcea py3: unskip tests/test_downloadermiddleware_httpcache.py and scrapy/downloadermiddlewares/httpcache.py 2016-01-15 15:41:31 +03:00
Konstantin Lopuhin 8330776c21 fix error reporting in test: we can fail in process_request too, so result should always be defined 2016-01-15 15:16:12 +03:00
Konstantin Lopuhin 4398d95a02 skip this file on py3 again - it has one compression test, sould be done separately 2016-01-15 14:54:12 +03:00
Konstantin Lopuhin ae6758ee97 Merge branch 'py3-http-downloaders' into py3-downloader-middleware 2016-01-15 14:39:27 +03:00
Konstantin Lopuhin a4ca1668d8 add https test for http10 handler (no luck with testing https with http11 so far) 2016-01-15 14:20:19 +03:00
Konstantin Lopuhin 96fcf4cea4 add a check that byte url is not accepted in http.Response on py3 2016-01-15 13:27:28 +03:00
Konstantin Lopuhin 87849780bc some py3 fixes for RFC2616Policy 2016-01-15 13:13:10 +03:00
Konstantin Lopuhin ddc91dda27 py3: fix _BaseTest in httpcache 2016-01-15 11:49:28 +03:00
Konstantin Lopuhin 94ab7bee6c py3: body to bytes in tests, unskip test file 2016-01-15 11:26:01 +03:00
Konstantin Lopuhin 99f1f2ad1d unskip tests and modules ported to py3 2016-01-14 19:00:48 +03:00
Konstantin Lopuhin 81a90c3af6 unskip part of test_download_gzip_response on py3, file a twisted issue for the remaining part 2016-01-14 18:47:06 +03:00
Konstantin Lopuhin 2aa6c92ffc py3 fixes in tests.mockserver 2016-01-14 17:52:50 +03:00
Konstantin Lopuhin f46a9d595d skip ftp tests on py3 for now 2016-01-14 17:31:58 +03:00
Konstantin Lopuhin 4950f5988e py3: pass http proxy tests 2016-01-14 17:24:08 +03:00
Konstantin Lopuhin c6f14a39de py3: fix http10 downloader - unicode host expected here 2016-01-14 16:50:16 +03:00
Konstantin Lopuhin 6b79fffa9a py3: pass all of HttpTestCase 2016-01-14 16:37:18 +03:00