RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,631 Commits 49 Branches 132 Tags 149 MiB
Commit Graph

5631 Commits

Author SHA1 Message Date
Pawel Miech dec04d50fa [image_pipeline] unify and simplify tests for setting loading 
there was identical test for different setting keys. I unified it into
one unit test.

Fixes comments for tests, adds comments about intention of uppercase attrs.

Adds another test for user defined setting keys and uppercase attrs.
 2016-07-13 17:02:12 +02:00
Pawel Miech c6277c9963 [image-pipeline] image settings with class name 
allow to have image settings with class name, so that settings for user defined ImagePipeline
subclasses can be defined easily.
 2016-07-13 17:02:12 +02:00
Pawel Miech 6bec0f773b [image_pipeline] tests for class attrs backward compatibility 
and docs about image pipeline settings.
 2016-07-13 17:02:12 +02:00
Pawel Miech 3fce2842ea [image_pipeline] bring back uppercase pipeline attributes 
allow users to have class attributes on image pipelines. This assumes
that class attributes are useful if users want to have different pipeline
classes inhriting from ImagePipeline.
 2016-07-13 17:02:12 +02:00
Paul Tremberth 308da48806 Merge pull request #2121 from redapple/backport-1.1-2094 
[backport][1.1] Catch and ignore certification verification exception for IP-address hosts (PR #2094)
 2016-07-13 16:46:15 +02:00
Paul Tremberth 1b7b87def1 Change wording of warning + docstring for ScrapyClientTLSOptions 2016-07-13 16:24:46 +02:00
Paul Tremberth 498719686c Rephrase warning 2016-07-13 16:24:46 +02:00
Paul Tremberth d2585704b8 Catch and ignore TLS verification exception for IP-address hosts 
Fixes GH-2092
 2016-07-13 16:24:46 +02:00
Paul Tremberth 0774b52245 Merge pull request #2112 from redapple/backport-1.1-pr2065 
[backport][1.1] Do not decompress binary/octet-stream responses (PR #2065)
 2016-07-11 13:37:08 +02:00
Paul Tremberth bda448f17d [HttpCompressionMiddleware] Do not decompress binary/octet-stream responses 2016-07-11 12:37:35 +02:00
Elias Dorneles a40f7c7a86 Merge pull request #2107 from redapple/backport-1.1-pr2052 
[backport][1.1] Enable genspider command outside project folder (PR #2052)
 2016-07-08 14:53:51 -03:00
Elias Dorneles 5b1531b077 Merge pull request #2106 from redapple/backport-1.1-pr2050 
[backport][1.1] Is_gzipped for application/x-gzip;charset=utf-8 (PR #2050)
 2016-07-08 14:53:41 -03:00
Valdir Stumm Junior 848bd4c2a2 document new genspider behavior 2016-07-08 18:48:39 +02:00
Valdir Stumm Junior e239246f4b remove references to Item classes in templates 2016-07-08 18:48:39 +02:00
Valdir Stumm Junior acb0c325aa enable genspider command outside projects 2016-07-08 18:48:39 +02:00
Joakim Uddholm 92bde95874 is_gzipped: Separated tests again. 2016-07-08 18:30:34 +02:00
Joakim Uddholm 658b6a6dcd is_gzipped: improved readability 2016-07-08 18:30:34 +02:00
Joakim Uddholm 27cbcbcdaf is_gzipped: Changed to regex to check the content-type header. Also suggested by @redapple. 2016-07-08 18:30:34 +02:00
Joakim Uddholm 346a376f5b Added new testcases suggested by @redapple. 2016-07-08 18:30:34 +02:00
Joakim Uddholm 5c511a29ec Separated tests based on case 2016-07-08 18:30:34 +02:00
Joakim Uddholm 59fda180d7 Suggested fix for is_gzipped 2016-07-08 18:30:34 +02:00
Joakim Uddholm 1c7ea7d686 Test to show bug with is_gzipped and Content-Type: application/gzip;charset. 2016-07-08 18:30:34 +02:00
Paul Tremberth b809907e0f Merge pull request #2042 from scrapy/backport-1.1-pr1974 
[backport][1.1] Retrying scrapy.core.downloader.handlers.http11.TunnelError (PR #1974)
 2016-07-08 14:11:28 +02:00
Paul Tremberth 3f03f01d80 Merge pull request #2043 from scrapy/backport-1.1-pr2008 
[backport][1.1] HttpCompressionMiddleware: don't decode HEAD responses (PR #2008)
 2016-07-08 14:10:52 +02:00
Paul Tremberth 8c937ed000 Merge pull request #2103 from redapple/backport-1.1-pr2038 
[backport][1.1] Do not fail on canonicalizing URLs with wrong netlocs (PR #2038)
 2016-07-08 12:46:31 +02:00
Paul Tremberth 53be5dc3e0 Do not fail on canonicalizing URLs with wrong netlocs 
Fixes #2010
 2016-07-08 12:35:56 +02:00
Paul Tremberth 0ef490e9ce Merge pull request #2102 from redapple/backport-1.1-pr2026 
[backport][1.1] Encourage descriptive PR titles (PR #2026)
 2016-07-08 12:31:47 +02:00
Νικόλαος-Διγενής Καραγιάννης 8ffb30fbef Encourage descriptive PR titles 2016-07-08 12:30:54 +02:00
Paul Tremberth 060a94b398 Merge pull request #2101 from redapple/backport-1.1-pr2001 
[backport][1.1] Fixed choosing of response class (PR #2001)
 2016-07-08 12:29:33 +02:00
Matvei Nazaruk eb0c1a3921 Added test for http11 choosing response type without content-type header. 2016-07-08 12:20:22 +02:00
Matvei Nazaruk 5e60d957e9 Fixed choosing of response class based on body. 2016-07-08 12:20:21 +02:00
Paul Tremberth 1c34e491d6 Merge pull request #2100 from redapple/backport-1.1-pr2069 
[backport][1.1] Add "Host" header in CONNECT requests to HTTPS proxies (PR #2069)
 2016-07-08 12:14:14 +02:00
Paul Tremberth 556fc2c334 Cleanup unused argument 2016-07-08 11:59:57 +02:00
Paul Tremberth d89f9df52e Fix CONNECT request timeout (with an ugly hack) 2016-07-08 11:59:56 +02:00
Paul Tremberth 99a93a0967 Add "Host" header in CONNECT requests to HTTPS proxies 2016-07-08 11:59:56 +02:00
Paul Tremberth cedf46c9bf Merge pull request #2099 from redapple/backport-1.1-pr2095 
[backport][1.1] TST pin pytest-cov to 2.2.1; upgrade pytest (PR #2095)
 2016-07-08 11:48:56 +02:00
Paul Tremberth c5b78f7712 Merge pull request #2098 from redapple/backport-1.1-pr2060 
[backport][1.1] [docs] Add note about Windows + Python3 (PR #2060)
 2016-07-08 11:22:53 +02:00
Mikhail Korobov cb8eccf603 TST pin pytest-cov to 2.2.1; upgrade pytest 2016-07-08 11:21:24 +02:00
Pawel Miech dc8db8b35e docs on Scrapy on Windows + Python 3 2016-07-08 11:17:04 +02:00
Pawel Miech adb837d7b7 [docs] warnings about windows + python 3 in faq and install 2016-07-08 11:16:52 +02:00
Pawel Miech d191785194 [docs] add note about windows + python3 2016-07-08 11:16:42 +02:00
Paul Tremberth 7b7e492d28 Do not commit on any HTML parsing speed assumption with BS4+lxml 2016-06-16 22:27:17 +02:00
Paul Tremberth 9db63f588d Add note on how to choose parser with BeautifulSoup 2016-06-16 22:27:04 +02:00
Paul Tremberth 05ce6afa3f Add FAQ entry on using BeautifulSoup in spider callbacks 2016-06-16 22:26:54 +02:00
Matt O'Connell 04d4cca545 Typo fixes 2016-06-16 22:21:41 +02:00
omer 06aee3f6b3 fix issue with '' in python 3 2016-06-08 17:46:22 +02:00
omer 9e3da69867 Do not decode of head response (2) 2016-06-08 17:46:06 +02:00
omer a9c73601ca Do not decode of head response 2016-06-08 17:45:56 +02:00
Pengyu CHEN c482f1923a Added: Retrying scrapy.core.downloader.handlers.http11.TunnelError 2016-06-08 17:41:30 +02:00
Paul Tremberth a04dbca22d Merge pull request #2041 from scrapy/backport-1.1-pr2015 
[backport][1.1] Use "url" variable in the example (PR #2015)
 2016-06-08 17:39:15 +02:00