RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,396 Commits 49 Branches 132 Tags 143 MiB
Commit Graph

6396 Commits

Author SHA1 Message Date
Mikhail Korobov 52a52e2388 Merge pull request #2001 from matveinazaruk/issue-2000 
[MRG+1] Fixed choosing of response class.
 2016-07-08 10:44:24 +06:00
Elias Dorneles 0ab7c1fbf9 Merge pull request #2052 from stummjr/genspider-standalone 
[MRG+1] Enable genspider command outside project folder
 2016-07-06 18:01:30 -03:00
Valdir Stumm Junior 081595a2e4 document new genspider behavior 2016-07-06 15:10:48 -03:00
Valdir Stumm Junior 8987b17730 remove references to Item classes in templates 2016-07-06 15:10:48 -03:00
Valdir Stumm Junior 1779f5feca enable genspider command outside projects 2016-07-06 15:10:48 -03:00
Mikhail Korobov 759a555d28 Merge pull request #2069 from redapple/https-connect-host 
[MRG] Add "Host" header in CONNECT requests to HTTPS proxies
 2016-07-06 21:43:41 +06:00
Paul Tremberth 1737c15826 Merge pull request #2091 from eLRuLL/default_headers_priority 
[MRG+1] prioritize default headers over user agent middlewares
 2016-07-06 17:29:39 +02:00
Paul Tremberth 15d0c89159 Cleanup unused argument 2016-07-06 17:15:21 +02:00
Mikhail Korobov 8ec6c22d60 Merge pull request #2095 from scrapy/testing-deps 
[MRG+1] TST pin pytest-cov to 2.2.1; upgrade pytest
 2016-07-06 20:07:19 +06:00
Mikhail Korobov 4273734744 TST pin pytest-cov to 2.2.1; upgrade pytest 2016-07-06 18:29:49 +05:00
Paul Tremberth 37efdde3e3 Catch and ignore TLS verification exception for IP-address hosts 
Fixes GH-2092
 2016-07-06 14:20:13 +02:00
Raul Gallegos 49ac7de231 prioritize default headers over user agent 2016-07-05 15:38:17 -05:00
Elias Dorneles 441df4c853 Merge pull request #2060 from pawelmhm/windows-python3 
[MRG+1] [docs] add note about windows + python3
 2016-07-05 10:26:33 -03:00
Νικόλαος-Διγενής Καραγιάννης 2a92ffb409 Encourage descriptive PR titles 2016-07-05 14:21:28 +03:00
Pawel Miech b67440dec0 docs on Scrapy on Windows + Python 3 2016-07-04 16:37:51 +02:00
Paul Tremberth 6539277f99 Fix CONNECT request timeout (with an ugly hack) 2016-06-21 17:14:41 +02:00
Paul Tremberth d9343463cb Add "Host" header in CONNECT requests to HTTPS proxies 2016-06-21 13:26:32 +02:00
Paul Tremberth 10a2c46e12 [HttpCompressionMiddleware] Do not decompress binary/octet-stream responses 2016-06-20 16:37:00 +02:00
Pawel Miech fa4d0cdfe5 [FilesPipeline, ImagesPipeline] fix for cls attrs with DEFAULT prefix 
some class attributes for ImagePipeline and FilesPipeline had DEFAULT prefix. These
attributes should be preserved as well, if users subclasses define values for
DEFAULT_<CLS_ATTRIBUTE_NAME> attribute this value should be preserved.
 2016-06-20 12:53:20 +02:00
Pawel Miech 07d1605586 [docs] warnings about windows + python 3 in faq and install 2016-06-17 13:28:51 +02:00
Pawel Miech 73cc066caa [docs] add note about windows + python3 2016-06-17 09:21:58 +02:00
Pawel Miech 10b79c9b3e [files-pipeline] update docs with note about settings 
for subclasses.
 2016-06-15 15:49:11 +02:00
Pawel Miech 539d34bce0 [media-pipeline, file-pipeline] allow setting custom settings for subclasses 
* move key_for_pipe function to media pipeline so that file pipeline can use it
* use key_for_pipe in file pipeline so that users can define custom settings for subclasses easily
* add tests for file pipelines attributes and settings
 2016-06-15 15:39:11 +02:00
Pawel Miech acbfdc6184 [files_pipeline] ensure class attributes are preserved 
dont override class attributes with default settings (same as in image pipeline).
 2016-06-15 15:14:28 +02:00
Pawel Miech c6d1686d98 [files_pipeline] unify tests for files pipeline 
if test tests same thing but for different field it can be unified into one.
 2016-06-15 14:48:25 +02:00
Pawel Miech 72e4d5f33e [image_pipeline] another test for subclass inheritance 
test case when subclass inherits from base class and has no attributes nor
settings defined.
 2016-06-15 14:07:17 +02:00
Pawel Miech ee39d11e45 [image_pipeline] refactor and simplify tests for image settings 
unify tests that test same thing for different attribute values into one. Add
better docstrings for tests.
 2016-06-15 11:25:38 +02:00
Joakim Uddholm 23f99e98c4 is_gzipped: Separated tests again. 2016-06-14 21:33:51 +02:00
Mikhail Korobov 80c296e091 Merge pull request #2048 from redapple/bs4-faq 
[MRG] Add FAQ entry on using BeautifulSoup in spider callbacks
 2016-06-14 23:45:04 +06:00
Paul Tremberth 1ff9a4828b Do not commit on any HTML parsing speed assumption with BS4+lxml 2016-06-14 19:33:56 +02:00
Pawel Miech d715172528 [image_pipeline] unify and simplify tests for setting loading 
there was identical test for different setting keys. I unified it into
one unit test.

Fixes comments for tests, adds comments about intention of uppercase attrs.

Adds another test for user defined setting keys and uppercase attrs.
 2016-06-14 19:09:56 +02:00
Joakim Uddholm 36928d897c is_gzipped: improved readability 2016-06-14 15:40:20 +02:00
Paul Tremberth 6cbd92fac1 Add note on how to choose parser with BeautifulSoup 2016-06-14 15:33:34 +02:00
Joakim Uddholm 259426ec99 is_gzipped: Changed to regex to check the content-type header. Also suggested by @redapple. 2016-06-14 14:39:16 +02:00
Joakim Uddholm 124e218a3b Added new testcases suggested by @redapple. 2016-06-14 14:22:18 +02:00
Elias Dorneles edca28326f Merge pull request #2054 from matt-oconnell/tutorial-typos 
Typo fixes
 2016-06-13 17:35:34 -03:00
Matvei Nazaruk b76b374648 Added test for http11 choosing response type without content-type header. 2016-06-13 23:21:38 +03:00
Matt O'Connell 0bb1288969 Typo fixes 2016-06-13 16:21:02 -04:00
Joakim Uddholm 2c98a88a0e Separated tests based on case 2016-06-12 10:49:34 +02:00
Joakim Uddholm db729f5b30 Suggested fix for is_gzipped 2016-06-12 02:26:16 +02:00
Joakim Uddholm 989f6b8843 Test to show bug with is_gzipped and Content-Type: application/gzip;charset. 2016-06-12 01:38:27 +02:00
Paul Tremberth 7978237ed5 Add FAQ entry on using BeautifulSoup in spider callbacks 2016-06-10 17:42:18 +02:00
Pawel Miech a62d4b081c [image-pipeline] image settings with class name 
allow to have image settings with class name, so that settings for user defined ImagePipeline
subclasses can be defined easily.
 2016-06-10 12:48:02 +02:00
Paul Tremberth 1aec5200bc Do not fail on canonicalizing URLs with wrong netlocs 
Fixes #2010
 2016-06-08 16:49:33 +02:00
Paul Tremberth b7925e4220 Merge pull request #2008 from foromer4/master 
[MRG+1] Fix issue HttpCompressionMiddleware tries to decode HEAD responses #1899
 2016-06-06 16:14:27 +02:00
Mikhail Korobov 6ff605b488 Merge pull request #1974 from starrify/more-errors-to-retry 
[MRG+1] Added: Retrying scrapy.core.downloader.handlers.http11.TunnelError
 2016-06-06 20:13:24 +06:00
Paul Tremberth f929853cc9 Merge pull request #2015 from lopuhin/patch-1 
[MRG+1] Use "url" variable in the example
 2016-06-06 16:11:29 +02:00
Konstantin Lopuhin da8d0eadd6 Use "url" variable in the example 
Instead of hardcoded http://www.example.com: without it url variable is unused and only one request will make it past dupefilter.
 2016-05-27 10:47:57 +03:00
omer c8ec79d959 fix issue with '' in python 3 2016-05-25 06:56:06 +03:00
Felipe Ruhland 9ad54b3815
Fix template description after create project 2016-05-24 13:03:33 -03:00