RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,396 Commits 49 Branches 132 Tags 176 MiB
Commit Graph

6396 Commits

Author SHA1 Message Date
harshasrinivas 694c6d3d74 Simplify retry_times assignment statement 2017-03-14 16:14:40 +05:30
Mikhail Korobov 4e4395f16f Merge pull request #2644 from delftswa2017/fsfilestore_fixme_fix 
[MRG+1] Fixed the FIXME in FSFilesStore
 2017-03-14 15:11:28 +05:00
Pawel Miech 0f2a5cdb8e [logformatter] 'flags' format spec backward compatibility 
pass 'flags' kwarg to logger so that it is compatible with old
format of CRAWLEDMSG.
 2017-03-13 15:16:38 +01:00
woxcab fbb411a805 Allowed passing objects of Mapping class or its subclass to the CaselessDict initializer 2017-03-13 14:16:39 +03:00
jorenham 3cd9185aa1 Fixed the FIXME; more specific exception catching 2017-03-12 23:02:07 +01:00
harshasrinivas 0d57b5cd43 Prevent max_retry_times override 2017-03-13 02:10:23 +05:30
Paul Tremberth 871134ee22 Refactor to also test FilesPipeline 2017-03-12 17:30:24 +01:00
harshasrinivas 810658bcc5 Add feature to set RETRY_TIMES per request (#2642) 2017-03-12 05:06:12 +05:30
Paul Tremberth 708f1b009b Add integration tests for MEDIA_ALLOW_REDIRECTS 2017-03-10 21:36:33 +01:00
Paul Tremberth 7dcc86e61a Add file listing resource + redirecting resource to MockServer 2017-03-10 21:35:25 +01:00
Paul Tremberth cfb56400b2 Merge pull request #2641 from redapple/release-notes-1.3.3-master 
Update release notes and versionadded for SPIDER_LOADER_WARN_ONLY
 2017-03-10 17:15:26 +01:00
Paul Tremberth a7f5207e9f Update version added for SPIDER_LOADER_WARN_ONLY 2017-03-10 16:38:09 +01:00
Paul Tremberth b2c505d8ec Set release date in changelog for v1.3.3 2017-03-10 16:34:21 +01:00
Paul Tremberth d8865b3304 Update changelog for upcoming 1.3.3 2017-03-10 16:34:21 +01:00
Mikhail Korobov 9c69e90056 Merge pull request #2632 from redapple/spider-loader-warn-or-fail 
[MRG] Add SPIDER_LOADER_WARN_ONLY to toggle between spiderloader failure or warning
 2017-03-09 23:01:27 +05:00
Paul Tremberth 9628a73972 Update settings docs for new SPIDER_LOADER_WARN_ONLY 2017-03-09 17:40:34 +01:00
Paul Tremberth f2ac24eb7b Do not only warn on wrong spider modules for "scrapy list" 2017-03-09 17:38:15 +01:00
Paul Tremberth 36160f1b0a Merge pull request #2477 from scrapy/recommend-anaconda-for-win 
[MRG+2] docs: installation instructions, mention conda in the beginning (closes #2475)
 2017-03-09 12:56:50 +01:00
Paul Tremberth c0cbaccb7b Merge pull request #2581 from lopuhin/respect-custom-log-level 
[MRG+1] Respect custom log level: fixes GH-1612
 2017-03-09 12:47:20 +01:00
Paul Tremberth db13ab5ec2 Merge pull request #2611 from delftswa2017/httpcachemiddleware_log 
[MRG+1] [logging] HttpCacheMiddleware: log cache directory at instantiation
 2017-03-09 12:44:10 +01:00
Paul Tremberth c5e193d079 Merge pull request #2636 from delftswa2017/contrib_docs_fix 
[MRG+1] Removed contrib section in contribution documentation
 2017-03-09 12:42:59 +01:00
Paul Tremberth 9cfe9ae098 Do not use self.assertRaises() as context manager 2017-03-09 12:21:03 +01:00
jorenham ac63d3a3cf Removed contrib section; contrib is deprecated 2017-03-09 10:56:23 +01:00
Paul Tremberth 7be773e14a Add SPIDER_LOADER_WARN_ONLY to toggle between spiderloader failure and warning 2017-03-07 17:40:40 +01:00
Mikhail Korobov d3f8f3d38a Merge pull request #2612 from redapple/dupe-spider-name-tests 
[WIP] Add warning on duplicate spider name
 2017-03-07 20:08:42 +05:00
Eugenio Lacuesta c7bb2fa8ce Feed exports: consistent and backwards compatible behaviour on indent 2017-03-07 11:56:00 -03:00
Eugenio Lacuesta 766b2c8453 Feed exports: enforce difference between None and 0 on indent 
Also rename params and settings from "indent_width" to just "indent"
 2017-03-07 11:56:00 -03:00
Eugenio Lacuesta 7e9153b38d Feed exports: beautify JSON and XML 2017-03-07 11:56:00 -03:00
Paul Tremberth 0b9a18e1a1 Warn only once for all spiders 2017-03-07 15:41:17 +01:00
Paul Tremberth 978306a223 Fix dupe spider name warning string tests 2017-03-07 14:48:16 +01:00
Paul Tremberth a017f7b932 Warn about modules where duplicate spider names were found 2017-03-07 14:44:27 +01:00
Mikhail Korobov 802ed30e8e Merge pull request #2391 from redapple/always-gunzip-content-enc-gzip 
[MRG] Always decompress Content-Encoding: gzip at HttpCompression stage
 2017-03-07 16:56:54 +05:00
Paul Tremberth b6378c7ef6 Revert to using self.assert methods 2017-03-07 12:28:24 +01:00
Konstantin Lopuhin 6f55ca4643 Revert unneeded test_crawl changes 2017-03-07 14:20:52 +03:00
Paul Tremberth 4caceccd59 Use 3-bytes for gzip archive type sniffing 2017-03-07 11:02:46 +01:00
Paul Tremberth b174744b80 Do not silently fail on gzip unzipping 2017-03-07 11:02:46 +01:00
Paul Tremberth 11cdf58abe Always decompress Content-Encoding: gzip at HttpCompression stage 
Let SitemapSpider handle decoding of .xml.gz files if necessary
 2017-03-07 11:02:46 +01:00
Mikhail Korobov 6320b743cf Merge pull request #2393 from redapple/response-replace-encoding 
[MRG] Use body to choose response type after decompression content
 2017-03-07 13:23:35 +05:00
Paul Tremberth e42b846a9f Use body to chose response type after decompression content 2017-03-06 23:10:38 +01:00
Mikhail Korobov 451f147468 Merge pull request #2628 from redapple/brotli-docs 
DOC Mention brotli support in HttpCompressionMiddleware section
 2017-03-06 22:16:21 +05:00
Mikhail Korobov 20c4f9f43a Merge pull request #2627 from redapple/referrer-empty-string-header 
Fix referrer policy from response headers and support explicit empty string
 2017-03-06 21:44:08 +05:00
Paul Tremberth 2a7d391e0b DOC Mention brotli support in HttpCompressionMiddleware section 2017-03-06 17:30:32 +01:00
Paul Tremberth 768f3155e5 Fix referrer policy from response headers and support explicit empty string 2017-03-06 16:20:37 +01:00
Paul Tremberth d7b26edf6b Merge pull request #2625 from redapple/release-notes-packaging-fix 
Update release notes for 1.0.7, 1.1.4 and 1.2.3
 2017-03-06 14:47:01 +01:00
Paul Tremberth a8b47d6c68 Update release notes for 1.0.7, 1.1.4 and 1.2.3 2017-03-06 14:25:52 +01:00
Paul Tremberth 0a17f9b55c Merge pull request #2334 from ArturGaspar/data_uri 
[MRG+1] data URI download handler.
 2017-03-06 13:50:46 +01:00
Mikhail Korobov 2c5fa0c130 Merge pull request #2617 from redapple/engine-dupe-statement 
Remove redundant slot.add_request() call in ExecutionEngine
 2017-03-03 21:44:07 +05:00
Paul Tremberth 30d812eea2 Remove redundant slot.add_request() call in ExecutionEngine 2017-03-03 17:15:37 +01:00
Paul Tremberth c68f99eed8 Refactor settings tests 2017-03-03 17:03:25 +01:00
Paul Tremberth c3b6feca0e Fix setting lookup for MEDIA_ALLOWED_REDIRECTS 2017-03-03 16:29:07 +01:00