RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,647 Commits 50 Branches 132 Tags 142 MiB
Commit Graph

481 Commits

Author SHA1 Message Date
Pawel Miech 0405750117 [FilesPipeline, ImagesPipeline] fix for cls attrs with DEFAULT prefix 
some class attributes for ImagePipeline and FilesPipeline had DEFAULT prefix. These
attributes should be preserved as well, if users subclasses define values for
DEFAULT_<CLS_ATTRIBUTE_NAME> attribute this value should be preserved.
 2016-07-13 17:02:13 +02:00
Pawel Miech 516618752b [media-pipeline, file-pipeline] allow setting custom settings for subclasses 
* move key_for_pipe function to media pipeline so that file pipeline can use it
* use key_for_pipe in file pipeline so that users can define custom settings for subclasses easily
* add tests for file pipelines attributes and settings
 2016-07-13 17:02:13 +02:00
Pawel Miech 0957f078bd [files_pipeline] ensure class attributes are preserved 
dont override class attributes with default settings (same as in image pipeline).
 2016-07-13 17:02:13 +02:00
Pawel Miech b3591eadaa [files_pipeline] unify tests for files pipeline 
if test tests same thing but for different field it can be unified into one.
 2016-07-13 17:02:13 +02:00
Pawel Miech 20a74dd27a [image_pipeline] another test for subclass inheritance 
test case when subclass inherits from base class and has no attributes nor
settings defined.
 2016-07-13 17:02:13 +02:00
Pawel Miech df4cda79d2 [image_pipeline] refactor and simplify tests for image settings 
unify tests that test same thing for different attribute values into one. Add
better docstrings for tests.
 2016-07-13 17:02:12 +02:00
Pawel Miech dec04d50fa [image_pipeline] unify and simplify tests for setting loading 
there was identical test for different setting keys. I unified it into
one unit test.

Fixes comments for tests, adds comments about intention of uppercase attrs.

Adds another test for user defined setting keys and uppercase attrs.
 2016-07-13 17:02:12 +02:00
Pawel Miech c6277c9963 [image-pipeline] image settings with class name 
allow to have image settings with class name, so that settings for user defined ImagePipeline
subclasses can be defined easily.
 2016-07-13 17:02:12 +02:00
Pawel Miech 6bec0f773b [image_pipeline] tests for class attrs backward compatibility 
and docs about image pipeline settings.
 2016-07-13 17:02:12 +02:00
Pawel Miech 3fce2842ea [image_pipeline] bring back uppercase pipeline attributes 
allow users to have class attributes on image pipelines. This assumes
that class attributes are useful if users want to have different pipeline
classes inhriting from ImagePipeline.
 2016-07-13 17:02:12 +02:00
Paul Tremberth d2585704b8 Catch and ignore TLS verification exception for IP-address hosts 
Fixes GH-2092
 2016-07-13 16:24:46 +02:00
Paul Tremberth bda448f17d [HttpCompressionMiddleware] Do not decompress binary/octet-stream responses 2016-07-11 12:37:35 +02:00
Elias Dorneles a40f7c7a86 Merge pull request #2107 from redapple/backport-1.1-pr2052 
[backport][1.1] Enable genspider command outside project folder (PR #2052)
 2016-07-08 14:53:51 -03:00
Valdir Stumm Junior acb0c325aa enable genspider command outside projects 2016-07-08 18:48:39 +02:00
Joakim Uddholm 92bde95874 is_gzipped: Separated tests again. 2016-07-08 18:30:34 +02:00
Joakim Uddholm 346a376f5b Added new testcases suggested by @redapple. 2016-07-08 18:30:34 +02:00
Joakim Uddholm 5c511a29ec Separated tests based on case 2016-07-08 18:30:34 +02:00
Joakim Uddholm 1c7ea7d686 Test to show bug with is_gzipped and Content-Type: application/gzip;charset. 2016-07-08 18:30:34 +02:00
Paul Tremberth 3f03f01d80 Merge pull request #2043 from scrapy/backport-1.1-pr2008 
[backport][1.1] HttpCompressionMiddleware: don't decode HEAD responses (PR #2008)
 2016-07-08 14:10:52 +02:00
Paul Tremberth 53be5dc3e0 Do not fail on canonicalizing URLs with wrong netlocs 
Fixes #2010
 2016-07-08 12:35:56 +02:00
Matvei Nazaruk eb0c1a3921 Added test for http11 choosing response type without content-type header. 2016-07-08 12:20:22 +02:00
Paul Tremberth d89f9df52e Fix CONNECT request timeout (with an ugly hack) 2016-07-08 11:59:56 +02:00
Mikhail Korobov cb8eccf603 TST pin pytest-cov to 2.2.1; upgrade pytest 2016-07-08 11:21:24 +02:00
omer 06aee3f6b3 fix issue with '' in python 3 2016-06-08 17:46:22 +02:00
omer a9c73601ca Do not decode of head response 2016-06-08 17:45:56 +02:00
Paul Tremberth a19f3475be Add idempotence tests for canonicalize_url 2016-04-27 13:55:07 +02:00
Paul Tremberth 7f94b57b3f Fix canonicalize_url() on Python 3 and re-enable tests 2016-04-27 13:54:36 +02:00
Paul Tremberth 08232a3f82 Use newer w3lib.url.safe_url_string() and re-enable HTTP request tests 2016-04-20 17:36:58 +02:00
Paul Tremberth 33a7caac90 Add HTTPS tests with non-hostname-maching server certificate 2016-04-20 17:25:25 +02:00
djunzu bb68cc120b Change FilesPipeline class attributes to instance attributes. 
modified:   scrapy/pipelines/files.py
	modified:   tests/test_pipeline_files.py
 2016-04-11 18:06:03 +02:00
djunzu 6627e05e92 Add tests. 
modified:   tests/test_pipeline_files.py
	modified:   tests/test_pipeline_images.py
 2016-04-11 18:06:02 +02:00
Paul Tremberth a1a799249b Add link extractor test for non-ASCII characters in query part of URL 2016-04-11 16:55:56 +02:00
Paul Tremberth 6672883df8 Fix link extractor tests for non-ASCII characters from latin1 document 
URL path component should use UTF-8 before percent-encoding (that's what
browsers do when you open scrapy/tests/sample_data/link_extractor/linkextractor_latin1.html
and follow the links)
This matches current w3lib v1.14.1
 2016-04-11 16:55:42 +02:00
Aron Bordin b2e914943d added FEED_TEMPDIR to settings 2016-04-04 13:59:04 +02:00
nyov 5ae8863d3d Rename isbinarytext function to binary_is_text for clarity 
Closes #1389
 2016-03-31 11:56:00 +02:00
pawelmhm b181c10fc4 response_status_message should not fail on non-standard HTTP codes 
utility is used in retry middleware and it was failing to handle non-standard HTTP codes.
Instead of raising exceptions when passing through to_native_str it should return
"Unknown status" message.
 2016-03-31 11:52:49 +02:00
Paul Tremberth a0c5c61855 Explicitly call Twisted transport stopProducing() on HTTP/1.1 timeouts 2016-03-01 15:00:44 +01:00
Paul Tremberth 6d4c51b266 Merge pull request #1809 from scrapy/backport-1.1-1787 
[backport][1.1] Better tracebacks
 2016-02-24 17:54:56 +01:00
Paul Tremberth e25052e4ef Add "Content-Length: 0" for body-less HTTP/1.1 POST requests 
GH-823 was fixed only for HTTP/1.0 (in GH-1089)
 2016-02-24 17:21:08 +01:00
Paul Tremberth 50f7b416c5 Minor change on quotes 
Trying to force Travis CI to build
 2016-02-24 17:14:48 +01:00
Mikhail Korobov 0d596c27a9 Preserve tracebacks better. Fixes GH-1760. 2016-02-24 17:14:48 +01:00
Mikhail Korobov 4459ee5d53 TST clean up RunSpiderCommandTest 2016-02-24 17:14:48 +01:00
Paul Tremberth a6bc758db1 Merge pull request #1792 from scrapy/backport-1.1-pr1761 
[backport][1.1.x] Py3 S3 botocore
 2016-02-19 11:17:20 +01:00
Paul Tremberth 3331cea788 More liberal Content-Disposition header parsing 
Fixes #1782
 2016-02-18 22:56:17 +01:00
Konstantin Lopuhin 25e481fd14 [backport][1.1.x] Py3 S3 botocore 
Originally from https://github.com/scrapy/scrapy/pull/1761
 2016-02-18 22:48:37 +01:00
Elias Dorneles f928014d91 add deprecation for pydispatch (thanks for the help @redapple) 2016-02-15 11:40:08 +01:00
Nicolas Pennequin 72229f92de MailSender.send: allow passing a charset. 
Resolves Issue #348
 2016-02-09 19:52:06 +01:00
orangain 281e9bd046 PY3: Implement some attributes of WrappedRequest required in Python 3 
This will fix #1770.
 2016-02-09 16:10:58 +01:00
orangain 522c71bb8a Fix SitemapSpider to extract sitemap urls from robots.txt properly 
This will fix #1766.
 2016-02-09 16:10:58 +01:00
Elias Dorneles a8a6f050e7 Merge pull request #1735 from ArturGaspar/master 
[MRG+1] Fix for KeyError in robots.txt middleware
 2016-02-03 13:30:06 -02:00