RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,293 Commits 49 Branches 132 Tags 145 MiB
Commit Graph

6293 Commits

Author SHA1 Message Date
Qiwei Huang 8ecc307e8f Update spiders.rst 
Added note to allowed_domain attribute with an example explaining what goes in the list
 2017-03-20 19:37:07 -07:00
Qiwei Huang c5f74f7d1a Update spiders.rst 
Added a note to allowed_domains attribute, reminding users not to add urls into the list.
 2017-03-20 18:52:33 -07:00
Mikhail Korobov ec55799d5e Merge pull request #2616 from redapple/mediapipeline-redirect-fix-continued 
[MRG] Allow redirections in media files downloads
 2017-03-16 15:31:28 +05:00
Paul Tremberth 1c0da1749d Merge pull request #2646 from woxcab/master 
[MRG+1] Allowed passing objects of Mapping class or its subclass to the CaselessDict initializer
 2017-03-15 13:11:58 +01:00
woxcab a84652e775 Init tests are split by initializer' input 2017-03-15 12:39:48 +03:00
Mikhail Korobov 4e4395f16f Merge pull request #2644 from delftswa2017/fsfilestore_fixme_fix 
[MRG+1] Fixed the FIXME in FSFilesStore
 2017-03-14 15:11:28 +05:00
woxcab fbb411a805 Allowed passing objects of Mapping class or its subclass to the CaselessDict initializer 2017-03-13 14:16:39 +03:00
jorenham 3cd9185aa1 Fixed the FIXME; more specific exception catching 2017-03-12 23:02:07 +01:00
Paul Tremberth 871134ee22 Refactor to also test FilesPipeline 2017-03-12 17:30:24 +01:00
Paul Tremberth 708f1b009b Add integration tests for MEDIA_ALLOW_REDIRECTS 2017-03-10 21:36:33 +01:00
Paul Tremberth 7dcc86e61a Add file listing resource + redirecting resource to MockServer 2017-03-10 21:35:25 +01:00
Paul Tremberth cfb56400b2 Merge pull request #2641 from redapple/release-notes-1.3.3-master 
Update release notes and versionadded for SPIDER_LOADER_WARN_ONLY
 2017-03-10 17:15:26 +01:00
Paul Tremberth a7f5207e9f Update version added for SPIDER_LOADER_WARN_ONLY 2017-03-10 16:38:09 +01:00
Paul Tremberth b2c505d8ec Set release date in changelog for v1.3.3 2017-03-10 16:34:21 +01:00
Paul Tremberth d8865b3304 Update changelog for upcoming 1.3.3 2017-03-10 16:34:21 +01:00
Mikhail Korobov 9c69e90056 Merge pull request #2632 from redapple/spider-loader-warn-or-fail 
[MRG] Add SPIDER_LOADER_WARN_ONLY to toggle between spiderloader failure or warning
 2017-03-09 23:01:27 +05:00
Paul Tremberth 9628a73972 Update settings docs for new SPIDER_LOADER_WARN_ONLY 2017-03-09 17:40:34 +01:00
Paul Tremberth f2ac24eb7b Do not only warn on wrong spider modules for "scrapy list" 2017-03-09 17:38:15 +01:00
Paul Tremberth 36160f1b0a Merge pull request #2477 from scrapy/recommend-anaconda-for-win 
[MRG+2] docs: installation instructions, mention conda in the beginning (closes #2475)
 2017-03-09 12:56:50 +01:00
Paul Tremberth c0cbaccb7b Merge pull request #2581 from lopuhin/respect-custom-log-level 
[MRG+1] Respect custom log level: fixes GH-1612
 2017-03-09 12:47:20 +01:00
Paul Tremberth db13ab5ec2 Merge pull request #2611 from delftswa2017/httpcachemiddleware_log 
[MRG+1] [logging] HttpCacheMiddleware: log cache directory at instantiation
 2017-03-09 12:44:10 +01:00
Paul Tremberth c5e193d079 Merge pull request #2636 from delftswa2017/contrib_docs_fix 
[MRG+1] Removed contrib section in contribution documentation
 2017-03-09 12:42:59 +01:00
Paul Tremberth 9cfe9ae098 Do not use self.assertRaises() as context manager 2017-03-09 12:21:03 +01:00
jorenham ac63d3a3cf Removed contrib section; contrib is deprecated 2017-03-09 10:56:23 +01:00
Paul Tremberth 7be773e14a Add SPIDER_LOADER_WARN_ONLY to toggle between spiderloader failure and warning 2017-03-07 17:40:40 +01:00
Mikhail Korobov d3f8f3d38a Merge pull request #2612 from redapple/dupe-spider-name-tests 
[WIP] Add warning on duplicate spider name
 2017-03-07 20:08:42 +05:00
Paul Tremberth 0b9a18e1a1 Warn only once for all spiders 2017-03-07 15:41:17 +01:00
Paul Tremberth 978306a223 Fix dupe spider name warning string tests 2017-03-07 14:48:16 +01:00
Paul Tremberth a017f7b932 Warn about modules where duplicate spider names were found 2017-03-07 14:44:27 +01:00
Mikhail Korobov 802ed30e8e Merge pull request #2391 from redapple/always-gunzip-content-enc-gzip 
[MRG] Always decompress Content-Encoding: gzip at HttpCompression stage
 2017-03-07 16:56:54 +05:00
Paul Tremberth b6378c7ef6 Revert to using self.assert methods 2017-03-07 12:28:24 +01:00
Konstantin Lopuhin 6f55ca4643 Revert unneeded test_crawl changes 2017-03-07 14:20:52 +03:00
Paul Tremberth 4caceccd59 Use 3-bytes for gzip archive type sniffing 2017-03-07 11:02:46 +01:00
Paul Tremberth b174744b80 Do not silently fail on gzip unzipping 2017-03-07 11:02:46 +01:00
Paul Tremberth 11cdf58abe Always decompress Content-Encoding: gzip at HttpCompression stage 
Let SitemapSpider handle decoding of .xml.gz files if necessary
 2017-03-07 11:02:46 +01:00
Mikhail Korobov 6320b743cf Merge pull request #2393 from redapple/response-replace-encoding 
[MRG] Use body to choose response type after decompression content
 2017-03-07 13:23:35 +05:00
Paul Tremberth e42b846a9f Use body to chose response type after decompression content 2017-03-06 23:10:38 +01:00
Mikhail Korobov 451f147468 Merge pull request #2628 from redapple/brotli-docs 
DOC Mention brotli support in HttpCompressionMiddleware section
 2017-03-06 22:16:21 +05:00
Mikhail Korobov 20c4f9f43a Merge pull request #2627 from redapple/referrer-empty-string-header 
Fix referrer policy from response headers and support explicit empty string
 2017-03-06 21:44:08 +05:00
Paul Tremberth 2a7d391e0b DOC Mention brotli support in HttpCompressionMiddleware section 2017-03-06 17:30:32 +01:00
Paul Tremberth 768f3155e5 Fix referrer policy from response headers and support explicit empty string 2017-03-06 16:20:37 +01:00
Paul Tremberth d7b26edf6b Merge pull request #2625 from redapple/release-notes-packaging-fix 
Update release notes for 1.0.7, 1.1.4 and 1.2.3
 2017-03-06 14:47:01 +01:00
Paul Tremberth a8b47d6c68 Update release notes for 1.0.7, 1.1.4 and 1.2.3 2017-03-06 14:25:52 +01:00
Paul Tremberth 0a17f9b55c Merge pull request #2334 from ArturGaspar/data_uri 
[MRG+1] data URI download handler.
 2017-03-06 13:50:46 +01:00
Mikhail Korobov 2c5fa0c130 Merge pull request #2617 from redapple/engine-dupe-statement 
Remove redundant slot.add_request() call in ExecutionEngine
 2017-03-03 21:44:07 +05:00
Paul Tremberth 30d812eea2 Remove redundant slot.add_request() call in ExecutionEngine 2017-03-03 17:15:37 +01:00
Paul Tremberth c68f99eed8 Refactor settings tests 2017-03-03 17:03:25 +01:00
Paul Tremberth c3b6feca0e Fix setting lookup for MEDIA_ALLOWED_REDIRECTS 2017-03-03 16:29:07 +01:00
Konstantin Lopuhin ef04cfd237 Respect log settings in custom_settings: fixes GH-1612 
A new root logger is installed when a crawler is created
if one was already installed before.
This allows to respect custom settings related to logging,
such as LOG_LEVEL, LOG_FILE, etc.
 2017-03-03 18:01:11 +03:00
Paul Tremberth f7e11b198e Cleanup 2017-03-03 16:00:59 +01:00