RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,809 Commits 49 Branches 132 Tags 148 MiB
Commit Graph

6809 Commits

Author SHA1 Message Date
Oto Brglez 605691792f Updating media-pipeline docs for S3-like storage. 2017-03-19 12:35:39 +01:00
harshasrinivas 10741aca72 Update docs - improve clarity 2017-03-19 06:17:28 +05:30
harshasrinivas 0d9ebd6e1e Update tests for max_retry_times 2017-03-19 06:15:46 +05:30
harshasrinivas 49c5afc5ff Fix bug involving OR condition 2017-03-19 06:08:35 +05:30
Pawel Miech 4345eaf1b6 [logformatter] backward compat comments 2017-03-17 08:11:20 +01:00
Mikhail Korobov ec55799d5e Merge pull request #2616 from redapple/mediapipeline-redirect-fix-continued 
[MRG] Allow redirections in media files downloads
 2017-03-16 15:31:28 +05:00
Paul Tremberth 1c0da1749d Merge pull request #2646 from woxcab/master 
[MRG+1] Allowed passing objects of Mapping class or its subclass to the CaselessDict initializer
 2017-03-15 13:11:58 +01:00
woxcab a84652e775 Init tests are split by initializer' input 2017-03-15 12:39:48 +03:00
Bernardas 7ba4b0a21b add support for embeded ptpython shell 2017-03-15 08:07:47 +00:00
harshasrinivas 9d97d788c0 Update docs for meta key 2017-03-15 04:13:47 +05:30
harshasrinivas e321ac9931 Update unittests for max_retry_times 2017-03-15 04:12:32 +05:30
harshasrinivas 966bd49c42 Update unittest for meta['max_retry_times'] 2017-03-14 16:23:47 +05:30
harshasrinivas 694c6d3d74 Simplify retry_times assignment statement 2017-03-14 16:14:40 +05:30
Mikhail Korobov 4e4395f16f Merge pull request #2644 from delftswa2017/fsfilestore_fixme_fix 
[MRG+1] Fixed the FIXME in FSFilesStore
 2017-03-14 15:11:28 +05:00
Pawel Miech 0f2a5cdb8e [logformatter] 'flags' format spec backward compatibility 
pass 'flags' kwarg to logger so that it is compatible with old
format of CRAWLEDMSG.
 2017-03-13 15:16:38 +01:00
woxcab fbb411a805 Allowed passing objects of Mapping class or its subclass to the CaselessDict initializer 2017-03-13 14:16:39 +03:00
jorenham 3cd9185aa1 Fixed the FIXME; more specific exception catching 2017-03-12 23:02:07 +01:00
harshasrinivas 0d57b5cd43 Prevent max_retry_times override 2017-03-13 02:10:23 +05:30
Paul Tremberth 871134ee22 Refactor to also test FilesPipeline 2017-03-12 17:30:24 +01:00
harshasrinivas 810658bcc5 Add feature to set RETRY_TIMES per request (#2642) 2017-03-12 05:06:12 +05:30
Paul Tremberth 708f1b009b Add integration tests for MEDIA_ALLOW_REDIRECTS 2017-03-10 21:36:33 +01:00
Paul Tremberth 7dcc86e61a Add file listing resource + redirecting resource to MockServer 2017-03-10 21:35:25 +01:00
Paul Tremberth cfb56400b2 Merge pull request #2641 from redapple/release-notes-1.3.3-master 
Update release notes and versionadded for SPIDER_LOADER_WARN_ONLY
 2017-03-10 17:15:26 +01:00
Paul Tremberth a7f5207e9f Update version added for SPIDER_LOADER_WARN_ONLY 2017-03-10 16:38:09 +01:00
Paul Tremberth b2c505d8ec Set release date in changelog for v1.3.3 2017-03-10 16:34:21 +01:00
Paul Tremberth d8865b3304 Update changelog for upcoming 1.3.3 2017-03-10 16:34:21 +01:00
Mikhail Korobov 9c69e90056 Merge pull request #2632 from redapple/spider-loader-warn-or-fail 
[MRG] Add SPIDER_LOADER_WARN_ONLY to toggle between spiderloader failure or warning
 2017-03-09 23:01:27 +05:00
Paul Tremberth 9628a73972 Update settings docs for new SPIDER_LOADER_WARN_ONLY 2017-03-09 17:40:34 +01:00
Paul Tremberth f2ac24eb7b Do not only warn on wrong spider modules for "scrapy list" 2017-03-09 17:38:15 +01:00
Paul Tremberth 36160f1b0a Merge pull request #2477 from scrapy/recommend-anaconda-for-win 
[MRG+2] docs: installation instructions, mention conda in the beginning (closes #2475)
 2017-03-09 12:56:50 +01:00
Paul Tremberth c0cbaccb7b Merge pull request #2581 from lopuhin/respect-custom-log-level 
[MRG+1] Respect custom log level: fixes GH-1612
 2017-03-09 12:47:20 +01:00
Paul Tremberth db13ab5ec2 Merge pull request #2611 from delftswa2017/httpcachemiddleware_log 
[MRG+1] [logging] HttpCacheMiddleware: log cache directory at instantiation
 2017-03-09 12:44:10 +01:00
Paul Tremberth c5e193d079 Merge pull request #2636 from delftswa2017/contrib_docs_fix 
[MRG+1] Removed contrib section in contribution documentation
 2017-03-09 12:42:59 +01:00
Paul Tremberth 9cfe9ae098 Do not use self.assertRaises() as context manager 2017-03-09 12:21:03 +01:00
jorenham ac63d3a3cf Removed contrib section; contrib is deprecated 2017-03-09 10:56:23 +01:00
Paul Tremberth 7be773e14a Add SPIDER_LOADER_WARN_ONLY to toggle between spiderloader failure and warning 2017-03-07 17:40:40 +01:00
Mikhail Korobov d3f8f3d38a Merge pull request #2612 from redapple/dupe-spider-name-tests 
[WIP] Add warning on duplicate spider name
 2017-03-07 20:08:42 +05:00
Eugenio Lacuesta c7bb2fa8ce Feed exports: consistent and backwards compatible behaviour on indent 2017-03-07 11:56:00 -03:00
Eugenio Lacuesta 766b2c8453 Feed exports: enforce difference between None and 0 on indent 
Also rename params and settings from "indent_width" to just "indent"
 2017-03-07 11:56:00 -03:00
Eugenio Lacuesta 7e9153b38d Feed exports: beautify JSON and XML 2017-03-07 11:56:00 -03:00
Paul Tremberth 0b9a18e1a1 Warn only once for all spiders 2017-03-07 15:41:17 +01:00
Paul Tremberth 978306a223 Fix dupe spider name warning string tests 2017-03-07 14:48:16 +01:00
Paul Tremberth a017f7b932 Warn about modules where duplicate spider names were found 2017-03-07 14:44:27 +01:00
Mikhail Korobov 802ed30e8e Merge pull request #2391 from redapple/always-gunzip-content-enc-gzip 
[MRG] Always decompress Content-Encoding: gzip at HttpCompression stage
 2017-03-07 16:56:54 +05:00
Paul Tremberth b6378c7ef6 Revert to using self.assert methods 2017-03-07 12:28:24 +01:00
Konstantin Lopuhin 6f55ca4643 Revert unneeded test_crawl changes 2017-03-07 14:20:52 +03:00
Paul Tremberth 4caceccd59 Use 3-bytes for gzip archive type sniffing 2017-03-07 11:02:46 +01:00
Paul Tremberth b174744b80 Do not silently fail on gzip unzipping 2017-03-07 11:02:46 +01:00
Paul Tremberth 11cdf58abe Always decompress Content-Encoding: gzip at HttpCompression stage 
Let SitemapSpider handle decoding of .xml.gz files if necessary
 2017-03-07 11:02:46 +01:00
Mikhail Korobov 6320b743cf Merge pull request #2393 from redapple/response-replace-encoding 
[MRG] Use body to choose response type after decompression content
 2017-03-07 13:23:35 +05:00