RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,937 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

547 Commits

Author SHA1 Message Date
Pawel Miech 72e4d5f33e [image_pipeline] another test for subclass inheritance 
test case when subclass inherits from base class and has no attributes nor
settings defined.
 2016-06-15 14:07:17 +02:00
Pawel Miech ee39d11e45 [image_pipeline] refactor and simplify tests for image settings 
unify tests that test same thing for different attribute values into one. Add
better docstrings for tests.
 2016-06-15 11:25:38 +02:00
Joakim Uddholm 23f99e98c4 is_gzipped: Separated tests again. 2016-06-14 21:33:51 +02:00
Pawel Miech d715172528 [image_pipeline] unify and simplify tests for setting loading 
there was identical test for different setting keys. I unified it into
one unit test.

Fixes comments for tests, adds comments about intention of uppercase attrs.

Adds another test for user defined setting keys and uppercase attrs.
 2016-06-14 19:09:56 +02:00
Joakim Uddholm 124e218a3b Added new testcases suggested by @redapple. 2016-06-14 14:22:18 +02:00
Matvei Nazaruk b76b374648 Added test for http11 choosing response type without content-type header. 2016-06-13 23:21:38 +03:00
Joakim Uddholm 2c98a88a0e Separated tests based on case 2016-06-12 10:49:34 +02:00
Joakim Uddholm 989f6b8843 Test to show bug with is_gzipped and Content-Type: application/gzip;charset. 2016-06-12 01:38:27 +02:00
Pawel Miech a62d4b081c [image-pipeline] image settings with class name 
allow to have image settings with class name, so that settings for user defined ImagePipeline
subclasses can be defined easily.
 2016-06-10 12:48:02 +02:00
Paul Tremberth 1aec5200bc Do not fail on canonicalizing URLs with wrong netlocs 
Fixes #2010
 2016-06-08 16:49:33 +02:00
omer c8ec79d959 fix issue with '' in python 3 2016-05-25 06:56:06 +03:00
Felipe Ruhland 9ad54b3815
Fix template description after create project 2016-05-24 13:03:33 -03:00
Felipe Ruhland b8a09d7ab7
Added tests for more or less parameters 2016-05-24 11:58:52 -03:00
Felipe Ruhland 2521f031d6
Created new tests for implementation 2016-05-23 23:15:53 -03:00
omer ce48bae533 Do not decode of head response 2016-05-23 19:52:38 +03:00
Pawel Miech 6c67db3917 [image_pipeline] tests for class attrs backward compatibility 
and docs about image pipeline settings.
 2016-05-18 12:04:52 +02:00
Pawel Miech 4cef1a1d00 [image_pipeline] bring back uppercase pipeline attributes 
allow users to have class attributes on image pipelines. This assumes
that class attributes are useful if users want to have different pipeline
classes inhriting from ImagePipeline.
 2016-05-13 12:35:35 +02:00
Paul Tremberth 0e11b3e6f0 Add idempotence tests for canonicalize_url 2016-04-26 20:03:17 +02:00
Paul Tremberth 68dedf54cb Fix canonicalize_url() on Python 3 and re-enable tests 2016-04-21 14:50:59 +02:00
Mikhail Korobov 73a5571c60 Merge pull request #1923 from scrapy/request-new-safe_url_string 
[MRG+1] Use newer w3lib.url.safe_url_string() and re-enable HTTP request tests
 2016-04-20 19:33:45 +06:00
Paul Tremberth cd979ace40 Add HTTPS tests with non-hostname-maching server certificate 2016-04-20 14:42:03 +02:00
Paul Tremberth d42a98d3b5 Use newer w3lib.url.safe_url_string() and re-enable HTTP request tests 2016-04-12 00:33:25 +02:00
Paul Tremberth 7b5243a263 Add link extractor test for non-ASCII characters in query part of URL 2016-04-09 15:15:01 +02:00
Paul Tremberth 1656fbcffa Fix link extractor tests for non-ASCII characters from latin1 document 
URL path component should use UTF-8 before percent-encoding (that's what
browsers do when you open scrapy/tests/sample_data/link_extractor/linkextractor_latin1.html
and follow the links)
This matches current w3lib v1.14.1
 2016-04-08 23:25:50 +02:00
Paul Tremberth 0ede017d2a Merge pull request #1891 from djunzu/update_files_images_pipelines 
[MRG+1] Change Files/ImagesPipelines class attributes to instance attributes
 2016-04-08 12:55:09 +02:00
Paul Tremberth bf7f675493 Merge pull request #1847 from aron-bordin/add_blocking_storage_path_setting 
[MRG+2] added BLOCKING_FEED_STORAGE_PATH to settings
 2016-04-01 15:47:06 +02:00
Aron Bordin 9250a5bffa added FEED_TEMPDIR to settings 2016-04-01 00:05:21 -03:00
djunzu 8228a0c491 Change FilesPipeline class attributes to instance attributes. 
modified:   scrapy/pipelines/files.py
	modified:   tests/test_pipeline_files.py
 2016-03-31 19:20:39 -03:00
djunzu e9d48f8a8e Add tests. 
modified:   tests/test_pipeline_files.py
	modified:   tests/test_pipeline_images.py
 2016-03-31 19:19:49 -03:00
Paul Tremberth 3ba5671fbc Merge pull request #1851 from nyov/binary_or_text 
[MRG+1] Rename isbinarytext function to binary_is_text for clarity
 2016-03-31 11:55:09 +02:00
nyov e8ca467572 Rename isbinarytext function to binary_is_text for clarity 
Closes #1389
 2016-03-30 15:44:15 +00:00
pawelmhm 65c7c05060 response_status_message should not fail on non-standard HTTP codes 
utility is used in retry middleware and it was failing to handle non-standard HTTP codes.
Instead of raising exceptions when passing through to_native_str it should return
"Unknown status" message.
 2016-03-12 14:16:40 +01:00
Paul Tremberth ecddc093a4 Explicitly call Twisted transport stopProducing() on HTTP/1.1 timeouts 2016-02-24 23:04:31 +01:00
Daniel Graña 513ba7a1fb Merge pull request #1800 from redapple/http11-post-content-length 
[MRG+1] Add "Content-Length: 0" for body-less HTTP/1.1 POST requests
 2016-02-23 15:00:33 -03:00
Elias Dorneles 10bcdb49b0 Merge pull request #1787 from scrapy/improve-errors 
[MRG+1] Better tracebacks
 2016-02-23 13:16:47 -03:00
Paul Tremberth 6174192564 Add "Content-Length: 0" for body-less HTTP/1.1 POST requests 
GH-823 was fixed only for HTTP/1.0 (in GH-1089)
 2016-02-20 23:27:04 +01:00
Paul Tremberth da36b7d386 Merge pull request #1761 from lopuhin/py3-s3-botocore 
[MRG+1] Py3 S3 botocore
 2016-02-18 15:11:03 +01:00
Paul Tremberth 104027d78d Minor change on quotes 
Trying to force Travis CI to build
 2016-02-18 11:45:03 +01:00
Konstantin Lopuhin d61fbcc8b5 Support headers in S3FilesStore.persist_file for botocore 2016-02-18 10:57:02 +03:00
Mikhail Korobov f766dd0ba8 Preserve tracebacks better. Fixes GH-1760. 2016-02-17 23:07:03 +05:00
Mikhail Korobov 06da7af9e2 TST clean up RunSpiderCommandTest 2016-02-17 23:03:12 +05:00
Paul Tremberth cabed6f183 More liberal Content-Disposition header parsing 
Fixes #1782
 2016-02-17 16:55:28 +01:00
Konstantin Lopuhin d1ecb8cd38 Fix S3TestCase for precise env: we reraise TypeError as NotConfigured in this case 2016-02-15 19:59:48 +03:00
Konstantin Lopuhin 77ebb13684 fix assertRaises for precise env 2016-02-15 19:59:48 +03:00
Konstantin Lopuhin 08bc41cc68 py3: reviewed s3 downloader handlers 2016-02-15 19:59:48 +03:00
Konstantin Lopuhin 3cb7a567ea py3 fix for TestS3FilesStore: checksum is a native string 2016-02-15 19:59:48 +03:00
Konstantin Lopuhin cfc567f48e botocore support for S3FilesStore 2016-02-15 19:59:48 +03:00
Konstantin Lopuhin 32cd8c9165 add direct test for S3FilesStore 2016-02-15 19:59:48 +03:00
Konstantin Lopuhin d1470e85a2 S3FeedStorageTest: pass on py3, add some non-ascii content to be sure 2016-02-15 19:59:48 +03:00
Konstantin Lopuhin 3ada45a9bb S3FeedStorageTest: add botocore support, and organize boto/botocore checks 2016-02-15 19:59:47 +03:00
Konstantin Lopuhin 5d2f067458 S3FeedStorageTest: delete key after test 2016-02-15 19:59:47 +03:00
Konstantin Lopuhin 19b2910ad1 Fix assert_aws_environ: check for botocore with boto fallback on PY2 2016-02-15 19:59:47 +03:00
Konstantin Lopuhin bcb92b50dc check that no extra kwargs are silently discarded 2016-02-15 19:59:47 +03:00
Konstantin Lopuhin d6bea3bf2e botocore not only does not allow passing our own Date header, but does not handle x-amz-date according to the spec 2016-02-15 19:59:47 +03:00
Konstantin Lopuhin 7748ee6bba mock date in s3 tests when using botocore 2016-02-15 19:59:47 +03:00
Konstantin Lopuhin c3fec83e7e use botocore by default, boto is still used in "precise" env 2016-02-15 19:59:47 +03:00
Konstantin Lopuhin 467553cc29 fix anon test: in this case we do no signing, just change the url 2016-02-15 19:59:47 +03:00
Konstantin Lopuhin eaf3a239e4 using botocore for s3 request signing: proof of concept 2016-02-15 19:59:46 +03:00
Paul Tremberth 41588397c0 Merge pull request #1765 from scrapy/add-deprecation-for-pydispatch 
[MRG+1] Add fallback and deprecation warning for pydispatch (fixes #1762)
 2016-02-11 19:29:53 +01:00
Elias Dorneles 164493df2e add deprecation for pydispatch (thanks for the help @redapple) 2016-02-11 16:15:28 -02:00
Paul Tremberth c083935806 Merge pull request #1771 from orangain/secure-cookies 
[MRG+1] PY3: Implement some attributes of WrappedRequest required in Python 3
 2016-02-08 11:52:00 +01:00
orangain 1f743996ff PY3: Implement some attributes of WrappedRequest required in Python 3 
This will fix #1770.
 2016-02-07 14:19:27 +09:00
orangain 25c56159b8 Fix SitemapSpider to extract sitemap urls from robots.txt properly 
This will fix #1766.
 2016-02-06 23:54:07 +09:00
Nicolas Pennequin 061c63592a MailSender.send: allow passing a charset. 
Resolves Issue #348
 2016-02-04 19:33:44 +01:00
Elias Dorneles a8a6f050e7 Merge pull request #1735 from ArturGaspar/master 
[MRG+1] Fix for KeyError in robots.txt middleware
 2016-02-03 13:30:06 -02:00
Mikhail Korobov 43a53aca12 Merge pull request #1746 from redapple/shell-settings-logging 
[MRG+1] Remove __str__ and __repr__ from settings, introduce copy_to_dict()
 2016-02-03 19:59:24 +05:00
stummjr bb2cf7c0d7 Fixed bug on XMLItemExporter with non-string fields in items 2016-01-30 10:00:06 -02:00
Paul Tremberth d843a0aae8 Amend "settings" command to output JSON for dict settings 2016-01-29 21:12:03 +01:00
Paul Tremberth aa78758bc7 Update tests for settings copy_to_dict() 2016-01-29 18:59:12 +01:00
Mikhail Korobov a35aec71e9 Merge pull request #1710 from redapple/1550-shell_file-cont 
[MRG+1] shell command's ability to open local files + tests
 2016-01-29 21:03:40 +05:00
Paul Tremberth 78f00401cd Remove unused import in tests 2016-01-29 16:56:05 +01:00
stummjr cf2ebb0687 Include tests for exporters: JSON, JSON-Lines, Pickle and Marshal. 2016-01-28 19:43:53 -02:00
Paul Tremberth c6f374f2eb Merge remote-tracking branch 'origin/master' into 1550-shell_file-cont 2016-01-28 14:02:48 +01:00
Paul Tremberth 481e251775 Move guess_scheme() tests to relevant test module 2016-01-28 13:51:50 +01:00
Paul Tremberth cae268402d Move guess_scheme() to scrapy.utils.url 2016-01-28 13:42:04 +01:00
stummjr c55ff110a3 Fix CSV exporter for non string Python types. 2016-01-27 15:43:17 -02:00
stummjr f1d971a5c0 fix PythonItemExporter for non-string types 2016-01-27 14:34:46 -02:00
Artur Gaspar dc8701ea42 Add test for already failed deferreds when downloading page in robots.txt middleware. 2016-01-27 12:56:42 -02:00
Mikhail Korobov 4bcbb77bcc response.text. Fixes GH-1729. 2016-01-27 01:28:11 +05:00
Paul Tremberth 29695375d1 Add test for raised exception with SpiderState extension when no JOBDIR used 2016-01-26 16:33:24 +01:00
Νικόλαος-Διγενής Καραγιάννης 1cffa99e0d tests+doc for subdomains in offsite middleware 2016-01-26 12:49:43 +02:00
Daniel Graña a7b86137d0 Merge pull request #1499 from scrapy/py3-port-exporters 
[MRG+1] PY3 exporters
 2016-01-26 01:01:30 -03:00
Daniel Graña d0eacfe0f9 Add test case for marshal item exporter 2016-01-26 00:26:27 -03:00
Elias Dorneles 2dfdde3c79 fallback to repr when can't convert to native string 2016-01-25 22:24:35 -02:00
Elias Dorneles 23b3336c1f add test for invalid option 2016-01-25 22:11:04 -02:00
Daniel Graña b20463183c Merge pull request #1716 from lopuhin/py3-test-pipeline-files-images 
[MRG+1] py3: test_pipeline_files and test_pipeline_images
 2016-01-25 18:24:12 -03:00
Elias Dorneles 0c44fac2b5 added tests for feed export marshal and pickle 2016-01-24 19:17:42 -02:00
Konstantin Lopuhin 1be90323c2 py3: properly skip s3 tests on py3 2016-01-24 23:44:56 +03:00
Konstantin Lopuhin 4233b3cda4 py3: reviewed passing test_spidermiddleware_httperror.py 2016-01-24 23:10:03 +03:00
Konstantin Lopuhin 097082cffa reviewed py3 compat in pipelines/images.py and pipelines/files.py 2016-01-24 23:05:23 +03:00
Konstantin Lopuhin 333d4c91fb py3: add boto to py3 test requirements, test_pipeline_files and test_pipeline_images passing now 2016-01-24 22:55:21 +03:00
Konstantin Lopuhin 860353b0c0 py3: unskip test_mail and scrapy/mail.py 2016-01-24 13:27:41 +03:00
Konstantin Lopuhin 9704226ee4 py3: fix test_mail - get_payload returns bytes when decode is True 2016-01-24 13:25:14 +03:00
Elias Dorneles 9fbe6f3e81 added feedexport test for xml output 2016-01-23 17:17:40 -02:00
Elias Dorneles c75f1fe46a restore bytes instead of text, for easier reviewing 2016-01-23 16:09:57 -02:00
Paul Tremberth 5f09da60c1 Revert "Use pytest.mark.parametrize decorator" 
This reverts commit 1a30a7774b.
 2016-01-22 23:48:58 +01:00
Paul Tremberth 1a30a7774b Use pytest.mark.parametrize decorator 2016-01-22 18:22:19 +01:00
Paul Tremberth 61ca8d587a Merge remote-tracking branch 'origin/master' into 1550-shell_file-cont 2016-01-22 18:16:36 +01:00
Paul Tremberth 7a51d370f3 Regex-based guess_scheme() + refactor tests 2016-01-22 17:16:27 +01:00
Elias Dorneles d0955fd083 add back test for latin-1 encoding 2016-01-22 10:07:55 -02:00