RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,624 Commits 49 Branches 132 Tags 132 MiB
Commit Graph

5624 Commits

Author SHA1 Message Date
Paul Tremberth d2585704b8 Catch and ignore TLS verification exception for IP-address hosts 
Fixes GH-2092
 2016-07-13 16:24:46 +02:00
Paul Tremberth 0774b52245 Merge pull request #2112 from redapple/backport-1.1-pr2065 
[backport][1.1] Do not decompress binary/octet-stream responses (PR #2065)
 2016-07-11 13:37:08 +02:00
Paul Tremberth bda448f17d [HttpCompressionMiddleware] Do not decompress binary/octet-stream responses 2016-07-11 12:37:35 +02:00
Elias Dorneles a40f7c7a86 Merge pull request #2107 from redapple/backport-1.1-pr2052 
[backport][1.1] Enable genspider command outside project folder (PR #2052)
 2016-07-08 14:53:51 -03:00
Elias Dorneles 5b1531b077 Merge pull request #2106 from redapple/backport-1.1-pr2050 
[backport][1.1] Is_gzipped for application/x-gzip;charset=utf-8 (PR #2050)
 2016-07-08 14:53:41 -03:00
Valdir Stumm Junior 848bd4c2a2 document new genspider behavior 2016-07-08 18:48:39 +02:00
Valdir Stumm Junior e239246f4b remove references to Item classes in templates 2016-07-08 18:48:39 +02:00
Valdir Stumm Junior acb0c325aa enable genspider command outside projects 2016-07-08 18:48:39 +02:00
Joakim Uddholm 92bde95874 is_gzipped: Separated tests again. 2016-07-08 18:30:34 +02:00
Joakim Uddholm 658b6a6dcd is_gzipped: improved readability 2016-07-08 18:30:34 +02:00
Joakim Uddholm 27cbcbcdaf is_gzipped: Changed to regex to check the content-type header. Also suggested by @redapple. 2016-07-08 18:30:34 +02:00
Joakim Uddholm 346a376f5b Added new testcases suggested by @redapple. 2016-07-08 18:30:34 +02:00
Joakim Uddholm 5c511a29ec Separated tests based on case 2016-07-08 18:30:34 +02:00
Joakim Uddholm 59fda180d7 Suggested fix for is_gzipped 2016-07-08 18:30:34 +02:00
Joakim Uddholm 1c7ea7d686 Test to show bug with is_gzipped and Content-Type: application/gzip;charset. 2016-07-08 18:30:34 +02:00
Paul Tremberth b809907e0f Merge pull request #2042 from scrapy/backport-1.1-pr1974 
[backport][1.1] Retrying scrapy.core.downloader.handlers.http11.TunnelError (PR #1974)
 2016-07-08 14:11:28 +02:00
Paul Tremberth 3f03f01d80 Merge pull request #2043 from scrapy/backport-1.1-pr2008 
[backport][1.1] HttpCompressionMiddleware: don't decode HEAD responses (PR #2008)
 2016-07-08 14:10:52 +02:00
Paul Tremberth 8c937ed000 Merge pull request #2103 from redapple/backport-1.1-pr2038 
[backport][1.1] Do not fail on canonicalizing URLs with wrong netlocs (PR #2038)
 2016-07-08 12:46:31 +02:00
Paul Tremberth 53be5dc3e0 Do not fail on canonicalizing URLs with wrong netlocs 
Fixes #2010
 2016-07-08 12:35:56 +02:00
Paul Tremberth 0ef490e9ce Merge pull request #2102 from redapple/backport-1.1-pr2026 
[backport][1.1] Encourage descriptive PR titles (PR #2026)
 2016-07-08 12:31:47 +02:00
Νικόλαος-Διγενής Καραγιάννης 8ffb30fbef Encourage descriptive PR titles 2016-07-08 12:30:54 +02:00
Paul Tremberth 060a94b398 Merge pull request #2101 from redapple/backport-1.1-pr2001 
[backport][1.1] Fixed choosing of response class (PR #2001)
 2016-07-08 12:29:33 +02:00
Matvei Nazaruk eb0c1a3921 Added test for http11 choosing response type without content-type header. 2016-07-08 12:20:22 +02:00
Matvei Nazaruk 5e60d957e9 Fixed choosing of response class based on body. 2016-07-08 12:20:21 +02:00
Paul Tremberth 1c34e491d6 Merge pull request #2100 from redapple/backport-1.1-pr2069 
[backport][1.1] Add "Host" header in CONNECT requests to HTTPS proxies (PR #2069)
 2016-07-08 12:14:14 +02:00
Paul Tremberth 556fc2c334 Cleanup unused argument 2016-07-08 11:59:57 +02:00
Paul Tremberth d89f9df52e Fix CONNECT request timeout (with an ugly hack) 2016-07-08 11:59:56 +02:00
Paul Tremberth 99a93a0967 Add "Host" header in CONNECT requests to HTTPS proxies 2016-07-08 11:59:56 +02:00
Paul Tremberth cedf46c9bf Merge pull request #2099 from redapple/backport-1.1-pr2095 
[backport][1.1] TST pin pytest-cov to 2.2.1; upgrade pytest (PR #2095)
 2016-07-08 11:48:56 +02:00
Paul Tremberth c5b78f7712 Merge pull request #2098 from redapple/backport-1.1-pr2060 
[backport][1.1] [docs] Add note about Windows + Python3 (PR #2060)
 2016-07-08 11:22:53 +02:00
Mikhail Korobov cb8eccf603 TST pin pytest-cov to 2.2.1; upgrade pytest 2016-07-08 11:21:24 +02:00
Pawel Miech dc8db8b35e docs on Scrapy on Windows + Python 3 2016-07-08 11:17:04 +02:00
Pawel Miech adb837d7b7 [docs] warnings about windows + python 3 in faq and install 2016-07-08 11:16:52 +02:00
Pawel Miech d191785194 [docs] add note about windows + python3 2016-07-08 11:16:42 +02:00
Paul Tremberth 7b7e492d28 Do not commit on any HTML parsing speed assumption with BS4+lxml 2016-06-16 22:27:17 +02:00
Paul Tremberth 9db63f588d Add note on how to choose parser with BeautifulSoup 2016-06-16 22:27:04 +02:00
Paul Tremberth 05ce6afa3f Add FAQ entry on using BeautifulSoup in spider callbacks 2016-06-16 22:26:54 +02:00
Matt O'Connell 04d4cca545 Typo fixes 2016-06-16 22:21:41 +02:00
omer 06aee3f6b3 fix issue with '' in python 3 2016-06-08 17:46:22 +02:00
omer 9e3da69867 Do not decode of head response (2) 2016-06-08 17:46:06 +02:00
omer a9c73601ca Do not decode of head response 2016-06-08 17:45:56 +02:00
Pengyu CHEN c482f1923a Added: Retrying scrapy.core.downloader.handlers.http11.TunnelError 2016-06-08 17:41:30 +02:00
Paul Tremberth a04dbca22d Merge pull request #2041 from scrapy/backport-1.1-pr2015 
[backport][1.1] Use "url" variable in the example (PR #2015)
 2016-06-08 17:39:15 +02:00
Konstantin Lopuhin e6a0baa8fe Use "url" variable in the example 
Instead of hardcoded http://www.example.com: without it url variable is unused and only one request will make it past dupefilter.
 2016-06-08 17:37:26 +02:00
Paul Tremberth 7415a41225 Merge pull request #2040 from scrapy/backport-1.1-pr1995 
[backport][1.1] Add info and example on errbacks (PR #1995)
 2016-06-08 17:34:16 +02:00
Paul Tremberth 1be1b454ef DOC Add info and example on errbacks 2016-06-08 17:32:56 +02:00
Paul Tremberth efaabbdb99 DOC Update copyright notice 2016-06-08 17:31:37 +02:00
Paul Tremberth e1277ff70b Merge pull request #2039 from scrapy/backport-1.1-pr1994 
[backport][1.1][DOC] Use idiomatic .extract_first() (PR #1994)
 2016-06-08 17:30:15 +02:00
Paul Tremberth 239254b282 [DOC][Overview] Use idiomatic .extract_first() 2016-06-08 17:27:55 +02:00
Paul Tremberth 4ea2453ae0 DOC Place FEED_TEMPDIR setting at lexicographical position 2016-06-08 17:24:33 +02:00