RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,222 Commits 42 Branches 132 Tags 161 MiB
Commit Graph

6222 Commits

Author SHA1 Message Date
Pawel Miech c6d1686d98 [files_pipeline] unify tests for files pipeline 
if test tests same thing but for different field it can be unified into one.
 2016-06-15 14:48:25 +02:00
Pawel Miech 72e4d5f33e [image_pipeline] another test for subclass inheritance 
test case when subclass inherits from base class and has no attributes nor
settings defined.
 2016-06-15 14:07:17 +02:00
Pawel Miech ee39d11e45 [image_pipeline] refactor and simplify tests for image settings 
unify tests that test same thing for different attribute values into one. Add
better docstrings for tests.
 2016-06-15 11:25:38 +02:00
Joakim Uddholm 23f99e98c4 is_gzipped: Separated tests again. 2016-06-14 21:33:51 +02:00
Mikhail Korobov 80c296e091 Merge pull request #2048 from redapple/bs4-faq 
[MRG] Add FAQ entry on using BeautifulSoup in spider callbacks
 2016-06-14 23:45:04 +06:00
Paul Tremberth 1ff9a4828b Do not commit on any HTML parsing speed assumption with BS4+lxml 2016-06-14 19:33:56 +02:00
Pawel Miech d715172528 [image_pipeline] unify and simplify tests for setting loading 
there was identical test for different setting keys. I unified it into
one unit test.

Fixes comments for tests, adds comments about intention of uppercase attrs.

Adds another test for user defined setting keys and uppercase attrs.
 2016-06-14 19:09:56 +02:00
Joakim Uddholm 36928d897c is_gzipped: improved readability 2016-06-14 15:40:20 +02:00
Paul Tremberth 6cbd92fac1 Add note on how to choose parser with BeautifulSoup 2016-06-14 15:33:34 +02:00
Joakim Uddholm 259426ec99 is_gzipped: Changed to regex to check the content-type header. Also suggested by @redapple. 2016-06-14 14:39:16 +02:00
Joakim Uddholm 124e218a3b Added new testcases suggested by @redapple. 2016-06-14 14:22:18 +02:00
Elias Dorneles edca28326f Merge pull request #2054 from matt-oconnell/tutorial-typos 
Typo fixes
 2016-06-13 17:35:34 -03:00
Matvei Nazaruk b76b374648 Added test for http11 choosing response type without content-type header. 2016-06-13 23:21:38 +03:00
Matt O'Connell 0bb1288969 Typo fixes 2016-06-13 16:21:02 -04:00
Joakim Uddholm 2c98a88a0e Separated tests based on case 2016-06-12 10:49:34 +02:00
Joakim Uddholm db729f5b30 Suggested fix for is_gzipped 2016-06-12 02:26:16 +02:00
Joakim Uddholm 989f6b8843 Test to show bug with is_gzipped and Content-Type: application/gzip;charset. 2016-06-12 01:38:27 +02:00
Paul Tremberth 7978237ed5 Add FAQ entry on using BeautifulSoup in spider callbacks 2016-06-10 17:42:18 +02:00
Pawel Miech a62d4b081c [image-pipeline] image settings with class name 
allow to have image settings with class name, so that settings for user defined ImagePipeline
subclasses can be defined easily.
 2016-06-10 12:48:02 +02:00
Paul Tremberth 1aec5200bc Do not fail on canonicalizing URLs with wrong netlocs 
Fixes #2010
 2016-06-08 16:49:33 +02:00
Paul Tremberth b7925e4220 Merge pull request #2008 from foromer4/master 
[MRG+1] Fix issue HttpCompressionMiddleware tries to decode HEAD responses #1899
 2016-06-06 16:14:27 +02:00
Mikhail Korobov 6ff605b488 Merge pull request #1974 from starrify/more-errors-to-retry 
[MRG+1] Added: Retrying scrapy.core.downloader.handlers.http11.TunnelError
 2016-06-06 20:13:24 +06:00
Paul Tremberth f929853cc9 Merge pull request #2015 from lopuhin/patch-1 
[MRG+1] Use "url" variable in the example
 2016-06-06 16:11:29 +02:00
Konstantin Lopuhin da8d0eadd6 Use "url" variable in the example 
Instead of hardcoded http://www.example.com: without it url variable is unused and only one request will make it past dupefilter.
 2016-05-27 10:47:57 +03:00
omer c8ec79d959 fix issue with '' in python 3 2016-05-25 06:56:06 +03:00
Felipe Ruhland 9ad54b3815
Fix template description after create project 2016-05-24 13:03:33 -03:00
Felipe Ruhland 85c4ecb928
Removed validation of project_name dir exists 2016-05-24 13:00:41 -03:00
Felipe Ruhland b8a09d7ab7
Added tests for more or less parameters 2016-05-24 11:58:52 -03:00
Felipe Ruhland fc9a45ee91
Simplified copytree function 2016-05-24 11:57:56 -03:00
Felipe Ruhland 24a45cc6e6
Fix py35 compatibility tests 2016-05-24 10:58:50 -03:00
omer ffa77e1ac6 Do not decode of head response (2) 2016-05-24 14:24:29 +03:00
Felipe Ruhland 089483aece
Updated docs for new option in command line 2016-05-23 23:16:15 -03:00
Felipe Ruhland 2521f031d6
Created new tests for implementation 2016-05-23 23:15:53 -03:00
Felipe Ruhland 6beb4f0119
Created project_dir optional parameter in startproject command line 2016-05-23 23:14:33 -03:00
omer ce48bae533 Do not decode of head response 2016-05-23 19:52:38 +03:00
Matvei Nazaruk 1bc9d35a87 Fixed choosing of response class based on body. 2016-05-19 22:24:37 +03:00
Elias Dorneles d3ced85ec3 Merge pull request #1995 from scrapy/docs-errback 
DOC Add info and example on errbacks
 2016-05-18 13:18:47 -03:00
Paul Tremberth b3367c7acd DOC Add info and example on errbacks 2016-05-18 18:00:09 +02:00
Paul Tremberth c2c8036a97 DOC Update copyright notice 2016-05-18 16:52:09 +02:00
Pawel Miech 6c67db3917 [image_pipeline] tests for class attrs backward compatibility 
and docs about image pipeline settings.
 2016-05-18 12:04:52 +02:00
Elias Dorneles ed4b9afbc3 Merge pull request #1994 from scrapy/overview-code 
[DOC][Overview] Use idiomatic .extract_first()
 2016-05-17 13:41:53 -03:00
Paul Tremberth 149c4cd41e [DOC][Overview] Use idiomatic .extract_first() 2016-05-17 17:53:40 +02:00
Paul Tremberth 9b3c72cb37 DOC Place FEED_TEMPDIR setting at lexicographical position 2016-05-17 16:25:57 +02:00
Pawel Miech 4cef1a1d00 [image_pipeline] bring back uppercase pipeline attributes 
allow users to have class attributes on image pipelines. This assumes
that class attributes are useful if users want to have different pipeline
classes inhriting from ImagePipeline.
 2016-05-13 12:35:35 +02:00
Paul Tremberth 0700e437f2 Update release notes and date for 1.1.0 2016-05-11 20:39:35 +02:00
mgachhui aecc23d24a Added Python 3.3+ to readme 2016-05-11 20:39:23 +02:00
Elias Dorneles 0e2b49c8c2 [MRG+1] Prevent empty warnings in case of certificate verification failures 
[MRG+1] Prevent empty warnings in case of certificate verification failures
 2016-05-11 10:51:45 -03:00
Paul Tremberth bd3e71a1ea Merge pull request #1980 from redapple/release-notes-1.0.6-merge 
Merge 1.0.6 release notes
 2016-05-11 14:50:08 +02:00
Paul Tremberth d62654dfda Merge 1.0.6 release notes 2016-05-11 14:39:55 +02:00
Paul Tremberth 137197b98b Prevent empty warnings in case of certificate verification failures 
Fixes #1976
 2016-05-11 13:12:35 +02:00