RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,658 Commits 49 Branches 132 Tags 144 MiB
Commit Graph

5658 Commits

Author SHA1 Message Date
Mikhail Korobov 759a555d28 Merge pull request #2069 from redapple/https-connect-host 
[MRG] Add "Host" header in CONNECT requests to HTTPS proxies
 2016-07-06 21:43:41 +06:00
Paul Tremberth 1737c15826 Merge pull request #2091 from eLRuLL/default_headers_priority 
[MRG+1] prioritize default headers over user agent middlewares
 2016-07-06 17:29:39 +02:00
Paul Tremberth 15d0c89159 Cleanup unused argument 2016-07-06 17:15:21 +02:00
Mikhail Korobov 8ec6c22d60 Merge pull request #2095 from scrapy/testing-deps 
[MRG+1] TST pin pytest-cov to 2.2.1; upgrade pytest
 2016-07-06 20:07:19 +06:00
Mikhail Korobov 4273734744 TST pin pytest-cov to 2.2.1; upgrade pytest 2016-07-06 18:29:49 +05:00
Raul Gallegos 49ac7de231 prioritize default headers over user agent 2016-07-05 15:38:17 -05:00
Elias Dorneles 441df4c853 Merge pull request #2060 from pawelmhm/windows-python3 
[MRG+1] [docs] add note about windows + python3
 2016-07-05 10:26:33 -03:00
Pawel Miech b67440dec0 docs on Scrapy on Windows + Python 3 2016-07-04 16:37:51 +02:00
Paul Tremberth 6539277f99 Fix CONNECT request timeout (with an ugly hack) 2016-06-21 17:14:41 +02:00
Paul Tremberth d9343463cb Add "Host" header in CONNECT requests to HTTPS proxies 2016-06-21 13:26:32 +02:00
Pawel Miech 07d1605586 [docs] warnings about windows + python 3 in faq and install 2016-06-17 13:28:51 +02:00
Pawel Miech 73cc066caa [docs] add note about windows + python3 2016-06-17 09:21:58 +02:00
Mikhail Korobov 80c296e091 Merge pull request #2048 from redapple/bs4-faq 
[MRG] Add FAQ entry on using BeautifulSoup in spider callbacks
 2016-06-14 23:45:04 +06:00
Paul Tremberth 1ff9a4828b Do not commit on any HTML parsing speed assumption with BS4+lxml 2016-06-14 19:33:56 +02:00
Paul Tremberth 6cbd92fac1 Add note on how to choose parser with BeautifulSoup 2016-06-14 15:33:34 +02:00
Elias Dorneles edca28326f Merge pull request #2054 from matt-oconnell/tutorial-typos 
Typo fixes
 2016-06-13 17:35:34 -03:00
Matt O'Connell 0bb1288969 Typo fixes 2016-06-13 16:21:02 -04:00
Paul Tremberth 7978237ed5 Add FAQ entry on using BeautifulSoup in spider callbacks 2016-06-10 17:42:18 +02:00
Paul Tremberth b7925e4220 Merge pull request #2008 from foromer4/master 
[MRG+1] Fix issue HttpCompressionMiddleware tries to decode HEAD responses #1899
 2016-06-06 16:14:27 +02:00
Mikhail Korobov 6ff605b488 Merge pull request #1974 from starrify/more-errors-to-retry 
[MRG+1] Added: Retrying scrapy.core.downloader.handlers.http11.TunnelError
 2016-06-06 20:13:24 +06:00
Paul Tremberth f929853cc9 Merge pull request #2015 from lopuhin/patch-1 
[MRG+1] Use "url" variable in the example
 2016-06-06 16:11:29 +02:00
Konstantin Lopuhin da8d0eadd6 Use "url" variable in the example 
Instead of hardcoded http://www.example.com: without it url variable is unused and only one request will make it past dupefilter.
 2016-05-27 10:47:57 +03:00
omer c8ec79d959 fix issue with '' in python 3 2016-05-25 06:56:06 +03:00
omer ffa77e1ac6 Do not decode of head response (2) 2016-05-24 14:24:29 +03:00
omer ce48bae533 Do not decode of head response 2016-05-23 19:52:38 +03:00
Elias Dorneles d3ced85ec3 Merge pull request #1995 from scrapy/docs-errback 
DOC Add info and example on errbacks
 2016-05-18 13:18:47 -03:00
Paul Tremberth b3367c7acd DOC Add info and example on errbacks 2016-05-18 18:00:09 +02:00
Paul Tremberth c2c8036a97 DOC Update copyright notice 2016-05-18 16:52:09 +02:00
Elias Dorneles ed4b9afbc3 Merge pull request #1994 from scrapy/overview-code 
[DOC][Overview] Use idiomatic .extract_first()
 2016-05-17 13:41:53 -03:00
Paul Tremberth 149c4cd41e [DOC][Overview] Use idiomatic .extract_first() 2016-05-17 17:53:40 +02:00
Paul Tremberth 9b3c72cb37 DOC Place FEED_TEMPDIR setting at lexicographical position 2016-05-17 16:25:57 +02:00
Paul Tremberth 0700e437f2 Update release notes and date for 1.1.0 2016-05-11 20:39:35 +02:00
mgachhui aecc23d24a Added Python 3.3+ to readme 2016-05-11 20:39:23 +02:00
Elias Dorneles 0e2b49c8c2 [MRG+1] Prevent empty warnings in case of certificate verification failures 
[MRG+1] Prevent empty warnings in case of certificate verification failures
 2016-05-11 10:51:45 -03:00
Paul Tremberth bd3e71a1ea Merge pull request #1980 from redapple/release-notes-1.0.6-merge 
Merge 1.0.6 release notes
 2016-05-11 14:50:08 +02:00
Paul Tremberth d62654dfda Merge 1.0.6 release notes 2016-05-11 14:39:55 +02:00
Paul Tremberth 137197b98b Prevent empty warnings in case of certificate verification failures 
Fixes #1976
 2016-05-11 13:12:35 +02:00
Paul Tremberth 6aa85aee2a Merge pull request #1975 from lipis/github 
Github >> GitHub
 2016-05-09 16:43:40 +02:00
Panayiotis Lipiridis 3f8be374c3 Github >> GitHub 2016-05-09 16:15:18 +02:00
Pengyu CHEN 334ee40e3d Added: Retrying scrapy.core.downloader.handlers.http11.TunnelError 2016-05-09 20:43:54 +08:00
Mikhail Korobov 064f05349f Merge pull request #1962 from redapple/w3lib-dep 
Bump w3lib version dependency in setup.py
 2016-04-29 16:55:10 +06:00
Paul Tremberth c8bff23751 Bump w3lib version dependency in setup.py 2016-04-29 10:29:37 +02:00
Paul Tremberth cb38f3896b Add notes on botocore support and JsonItemExporter opening/closing brackets 2016-04-28 22:34:17 +02:00
Paul Tremberth dc9de7524f Update release notes with recent features and bug fixes. 2016-04-28 22:34:07 +02:00
Mikhail Korobov dbef7e2b45 Merge pull request #1947 from scrapy/canonicalize-url 
[MRG+1] Fix canonicalize_url() on Python 3 and re-enable tests
 2016-04-27 01:32:42 +06:00
Paul Tremberth 0e11b3e6f0 Add idempotence tests for canonicalize_url 2016-04-26 20:03:17 +02:00
Paul Tremberth efbe75ea5e Use six.PY2 also for conditional imports 2016-04-26 16:14:15 +02:00
Paul Tremberth 25401fd322 Use six.PY2 instead of six.PY3 for Python version variations 
Also don't test passed encoding against 'utf8';
Just consider that if encoding failed, it must have been another encoding.
 2016-04-26 15:12:00 +02:00
Paul Tremberth cf716ea200 Merge pull request #1950 from patcon/patch-1 
[MRG+1] Place brackets on own lines with JsonItemExporter
 2016-04-25 15:21:21 +02:00
Patrick Connolly 19a4a0adf9 Place brackets on own lines with JsonItemExporter. 
Placing the opening and closing brackets on their own lines makes it slightly easier to sort lines after the `spider_closed` signal is fired.
 2016-04-24 14:55:08 -04:00