RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,719 Commits 49 Branches 132 Tags 182 MiB
Commit Graph

515 Commits

Author SHA1 Message Date
Paul Tremberth ec1c61504a Merge pull request #2005 from feliperuhland/master 
[MRG+1] Included new optional parameter in startproject command line
 2016-07-19 12:31:06 +02:00
Mikhail Korobov 79639d0fec Merge pull request #1989 from pawelmhm/fix-images-pipeline-uppercase-other 
[MRG+1] [image_pipeline] bring back uppercase class attributes
 2016-07-13 14:44:00 +00:00
Mikhail Korobov 2dd1a9e3bc Merge pull request #2094 from redapple/dns-invalid-id 
Catch and ignore certification verification exception for IP-address hosts
 2016-07-13 10:48:08 +00:00
Paul Tremberth c3109daa72 Merge pull request #2034 from dracony/master 
[MRG+1] Added option to turn off ensure_ascii for JSON exporters
 2016-07-12 17:01:09 +02:00
Dracony 33a39b368f added FEED_EXPORT_ENCODING setting to allow encoding specification 2016-07-12 16:20:17 +02:00
Paul Tremberth 778f1cf84c Merge remote-tracking branch 'origin/master' into octet-stream-no-decompress 2016-07-08 18:13:20 +02:00
Elias Dorneles d43a35735a Merge pull request #2050 from Tethik/is_gzipped_fix 
[MRG+1] Is_gzipped for application/x-gzip;charset=utf-8
 2016-07-08 08:47:56 -03:00
Mikhail Korobov b7553d921a Merge pull request #2038 from redapple/canonicalize-idna-failures 
[MRG] Do not fail on canonicalizing URLs with wrong netlocs
 2016-07-08 10:47:54 +06:00
Mikhail Korobov 52a52e2388 Merge pull request #2001 from matveinazaruk/issue-2000 
[MRG+1] Fixed choosing of response class.
 2016-07-08 10:44:24 +06:00
Valdir Stumm Junior 1779f5feca enable genspider command outside projects 2016-07-06 15:10:48 -03:00
Mikhail Korobov 759a555d28 Merge pull request #2069 from redapple/https-connect-host 
[MRG] Add "Host" header in CONNECT requests to HTTPS proxies
 2016-07-06 21:43:41 +06:00
Mikhail Korobov 4273734744 TST pin pytest-cov to 2.2.1; upgrade pytest 2016-07-06 18:29:49 +05:00
Paul Tremberth 37efdde3e3 Catch and ignore TLS verification exception for IP-address hosts 
Fixes GH-2092
 2016-07-06 14:20:13 +02:00
Paul Tremberth 6539277f99 Fix CONNECT request timeout (with an ugly hack) 2016-06-21 17:14:41 +02:00
Paul Tremberth 10a2c46e12 [HttpCompressionMiddleware] Do not decompress binary/octet-stream responses 2016-06-20 16:37:00 +02:00
Pawel Miech fa4d0cdfe5 [FilesPipeline, ImagesPipeline] fix for cls attrs with DEFAULT prefix 
some class attributes for ImagePipeline and FilesPipeline had DEFAULT prefix. These
attributes should be preserved as well, if users subclasses define values for
DEFAULT_<CLS_ATTRIBUTE_NAME> attribute this value should be preserved.
 2016-06-20 12:53:20 +02:00
Pawel Miech 539d34bce0 [media-pipeline, file-pipeline] allow setting custom settings for subclasses 
* move key_for_pipe function to media pipeline so that file pipeline can use it
* use key_for_pipe in file pipeline so that users can define custom settings for subclasses easily
* add tests for file pipelines attributes and settings
 2016-06-15 15:39:11 +02:00
Pawel Miech acbfdc6184 [files_pipeline] ensure class attributes are preserved 
dont override class attributes with default settings (same as in image pipeline).
 2016-06-15 15:14:28 +02:00
Pawel Miech c6d1686d98 [files_pipeline] unify tests for files pipeline 
if test tests same thing but for different field it can be unified into one.
 2016-06-15 14:48:25 +02:00
Pawel Miech 72e4d5f33e [image_pipeline] another test for subclass inheritance 
test case when subclass inherits from base class and has no attributes nor
settings defined.
 2016-06-15 14:07:17 +02:00
Pawel Miech ee39d11e45 [image_pipeline] refactor and simplify tests for image settings 
unify tests that test same thing for different attribute values into one. Add
better docstrings for tests.
 2016-06-15 11:25:38 +02:00
Joakim Uddholm 23f99e98c4 is_gzipped: Separated tests again. 2016-06-14 21:33:51 +02:00
Pawel Miech d715172528 [image_pipeline] unify and simplify tests for setting loading 
there was identical test for different setting keys. I unified it into
one unit test.

Fixes comments for tests, adds comments about intention of uppercase attrs.

Adds another test for user defined setting keys and uppercase attrs.
 2016-06-14 19:09:56 +02:00
Joakim Uddholm 124e218a3b Added new testcases suggested by @redapple. 2016-06-14 14:22:18 +02:00
Matvei Nazaruk b76b374648 Added test for http11 choosing response type without content-type header. 2016-06-13 23:21:38 +03:00
Joakim Uddholm 2c98a88a0e Separated tests based on case 2016-06-12 10:49:34 +02:00
Joakim Uddholm 989f6b8843 Test to show bug with is_gzipped and Content-Type: application/gzip;charset. 2016-06-12 01:38:27 +02:00
Pawel Miech a62d4b081c [image-pipeline] image settings with class name 
allow to have image settings with class name, so that settings for user defined ImagePipeline
subclasses can be defined easily.
 2016-06-10 12:48:02 +02:00
Paul Tremberth 1aec5200bc Do not fail on canonicalizing URLs with wrong netlocs 
Fixes #2010
 2016-06-08 16:49:33 +02:00
omer c8ec79d959 fix issue with '' in python 3 2016-05-25 06:56:06 +03:00
Felipe Ruhland 9ad54b3815
Fix template description after create project 2016-05-24 13:03:33 -03:00
Felipe Ruhland b8a09d7ab7
Added tests for more or less parameters 2016-05-24 11:58:52 -03:00
Felipe Ruhland 2521f031d6
Created new tests for implementation 2016-05-23 23:15:53 -03:00
omer ce48bae533 Do not decode of head response 2016-05-23 19:52:38 +03:00
Pawel Miech 6c67db3917 [image_pipeline] tests for class attrs backward compatibility 
and docs about image pipeline settings.
 2016-05-18 12:04:52 +02:00
Pawel Miech 4cef1a1d00 [image_pipeline] bring back uppercase pipeline attributes 
allow users to have class attributes on image pipelines. This assumes
that class attributes are useful if users want to have different pipeline
classes inhriting from ImagePipeline.
 2016-05-13 12:35:35 +02:00
Paul Tremberth 0e11b3e6f0 Add idempotence tests for canonicalize_url 2016-04-26 20:03:17 +02:00
Paul Tremberth 68dedf54cb Fix canonicalize_url() on Python 3 and re-enable tests 2016-04-21 14:50:59 +02:00
Mikhail Korobov 73a5571c60 Merge pull request #1923 from scrapy/request-new-safe_url_string 
[MRG+1] Use newer w3lib.url.safe_url_string() and re-enable HTTP request tests
 2016-04-20 19:33:45 +06:00
Paul Tremberth cd979ace40 Add HTTPS tests with non-hostname-maching server certificate 2016-04-20 14:42:03 +02:00
Paul Tremberth d42a98d3b5 Use newer w3lib.url.safe_url_string() and re-enable HTTP request tests 2016-04-12 00:33:25 +02:00
Paul Tremberth 7b5243a263 Add link extractor test for non-ASCII characters in query part of URL 2016-04-09 15:15:01 +02:00
Paul Tremberth 1656fbcffa Fix link extractor tests for non-ASCII characters from latin1 document 
URL path component should use UTF-8 before percent-encoding (that's what
browsers do when you open scrapy/tests/sample_data/link_extractor/linkextractor_latin1.html
and follow the links)
This matches current w3lib v1.14.1
 2016-04-08 23:25:50 +02:00
Paul Tremberth 0ede017d2a Merge pull request #1891 from djunzu/update_files_images_pipelines 
[MRG+1] Change Files/ImagesPipelines class attributes to instance attributes
 2016-04-08 12:55:09 +02:00
Paul Tremberth bf7f675493 Merge pull request #1847 from aron-bordin/add_blocking_storage_path_setting 
[MRG+2] added BLOCKING_FEED_STORAGE_PATH to settings
 2016-04-01 15:47:06 +02:00
Aron Bordin 9250a5bffa added FEED_TEMPDIR to settings 2016-04-01 00:05:21 -03:00
djunzu 8228a0c491 Change FilesPipeline class attributes to instance attributes. 
modified:   scrapy/pipelines/files.py
	modified:   tests/test_pipeline_files.py
 2016-03-31 19:20:39 -03:00
djunzu e9d48f8a8e Add tests. 
modified:   tests/test_pipeline_files.py
	modified:   tests/test_pipeline_images.py
 2016-03-31 19:19:49 -03:00
Paul Tremberth 3ba5671fbc Merge pull request #1851 from nyov/binary_or_text 
[MRG+1] Rename isbinarytext function to binary_is_text for clarity
 2016-03-31 11:55:09 +02:00
nyov e8ca467572 Rename isbinarytext function to binary_is_text for clarity 
Closes #1389
 2016-03-30 15:44:15 +00:00