RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,561 Commits 49 Branches 132 Tags 179 MiB
Commit Graph

5561 Commits

Author SHA1 Message Date
Konstantin Lopuhin fc8cd45a48 Fix a race condition in the FilesPipeline 
Checksum calculation could happen simultaniously with
persisting the file in the store (which is done in a thread):
they operated on the same buf object.
Concretely this lead to a bug with S3FilesStore
when using botocore: the signature did not match because
the position in the buf was already at the end.
The fix is to move checksum calculation before passing buf
to the store.
 2016-03-27 21:56:47 +02:00
Konstantin Lopuhin 5045a4f168 Fix handling of meta=None in S3FilesStore.persist_file 2016-03-25 18:35:55 +03:00
Mikhail Korobov ebef6d7c6d Merge pull request #1848 from aron-bordin/small_doc_style_fixes 
small doc style fixes
 2016-03-07 08:49:25 +05:00
Aron Bordin 2cfe9e424d small doc style fixes 2016-03-05 19:54:06 -03:00
Paul Tremberth e122c569fe Merge pull request #1842 from nyov/nyov/docs 
[MRG+1] Update documentation links
 2016-03-04 11:50:26 +01:00
nyov 5876b9aa30 Update documentation links 2016-03-03 16:28:33 +00:00
Paul Tremberth 9f4fe5dc4a Merge pull request #1822 from nyov/nyov/scheduler 
[MRG+1] Allow core Scheduler priority queue customization
 2016-03-02 14:20:40 +01:00
Mikhail Korobov 6b2871dadd Merge pull request #1835 from djunzu/add_pps_to_IGNORED_EXTENSIONS 
[MRG+1] Add pps extension to IGNORED_EXTENSIONS
 2016-03-02 16:15:51 +05:00
djunzu 0e288d4a71 Add pps extension to IGNORED_EXTENSIONS 
modified:   scrapy/linkextractors/__init__.py
 2016-03-01 21:02:13 -03:00
nyov 2a6524ee3a Allow core Scheduler priority queue customization 2016-03-01 13:58:40 +00:00
Daniel Graña b8fcb46e67 Merge pull request #1804 from redapple/enable-test-dwnld-timeout 
Re-enable HTTPS tests for download timeouts
 2016-03-01 10:44:24 -03:00
Daniel Graña 21da493109 Merge pull request #1828 from scrapy/py3-classifiers 
[MRG+1] declare Python 3 support in setup.py
 2016-03-01 10:34:36 -03:00
Daniel Graña cf535fe840 Merge pull request #1827 from scrapy/proxy-auth-test 
[MRG+1] Extract a function to build CONNECT request; add tests for it.
 2016-03-01 10:34:24 -03:00
Mikhail Korobov 17d3bec699 declare Python 3 support in setup.py 2016-03-01 16:34:13 +05:00
Mikhail Korobov 94e28adfb7 Extract a function to build CONNECT request; add tests for it. See GH-1701 and GH-1808. 2016-03-01 16:29:12 +05:00
Mikhail Korobov e8635cd03c Merge pull request #1826 from redapple/universal-wheels 
Build universal wheels
 2016-03-01 15:23:57 +05:00
Paul Tremberth f3c7a509b9 Build universal wheels 2016-03-01 11:00:20 +01:00
Mikhail Korobov 16512a32ea Merge pull request #1825 from redapple/relnotes-s3policy 
[MRG+1] Update release notes about change of default S3 ACL policy to "private"
 2016-02-29 17:02:18 +05:00
Paul Tremberth 90c64a6d13 Update release notes about change of default S3 ACL policy to "private" 2016-02-29 12:27:25 +01:00
Paul Tremberth 74158611c8 Merge pull request #1818 from lagenar/master 
[MRG+1] Refactored SpiderLoader class constructor for easier subclassing
 2016-02-26 23:23:34 +01:00
Paul Tremberth 241ae9f22f Merge pull request #1820 from redapple/http-tls-settings 
[MRG+1] Document DOWNLOADER_* settings for HTTP/1.0 and TLS
 2016-02-26 23:20:33 +01:00
Paul Tremberth 709b4fa884 Update release notes about HTTPS downloader 2016-02-26 18:35:29 +01:00
Paul Tremberth 174f526709 Document DOWNLOADER_* settings for HTTP/1.0 and TLS 2016-02-26 18:23:57 +01:00
Lucas Moauro 1ce4c86c2b Refactored SpiderLoader class constructor for easier subclassing 2016-02-25 21:08:50 -03:00
Paul Tremberth 84dea19455 Update release notes 2016-02-25 00:59:15 +01:00
Paul Tremberth c9e781357d Explicitly call Twisted transport stopProducing() on HTTP/1.0 timeouts 2016-02-24 23:04:31 +01:00
Paul Tremberth ecddc093a4 Explicitly call Twisted transport stopProducing() on HTTP/1.1 timeouts 2016-02-24 23:04:31 +01:00
Mikhail Korobov f7a48b0c6b Merge pull request #1794 from redapple/twisted-tls 
[MRG+1] Use best practices for TLS connections when using Twisted>=14.0
 2016-02-25 02:23:33 +05:00
Paul Tremberth 4e93501304 Merge pull request #1796 from lopuhin/s3-acl-private 
[MRG+1] Change default S3 ACL to "private" and allow customization via settings
 2016-02-24 22:18:43 +01:00
Paul Tremberth 0336c2504a Use context factory class name in warning message 2016-02-24 16:42:25 +01:00
Paul Tremberth c29a1b98b3 Make warning message a bit nicer in logs 2016-02-24 16:01:18 +01:00
Konstantin Lopuhin 6137dd96d9 Fix documentation for S3_STORE_ACL (now settings.FILES_STORE_S3_ACL) settings: it has nothing to do with feed exporters. 2016-02-24 10:16:41 +03:00
Konstantin Lopuhin 164f300762 See #1778 - change default S3 ACL to "private" and allow customization via settings 2016-02-24 10:16:41 +03:00
Paul Tremberth c9890d5f76 Add warning for context factories not accepting `method` param 2016-02-24 01:26:04 +01:00
Paul Tremberth 095495e99c Backward-compatibility for common Scrapy context factory patterns 2016-02-24 01:24:58 +01:00
Daniel Graña 513ba7a1fb Merge pull request #1800 from redapple/http11-post-content-length 
[MRG+1] Add "Content-Length: 0" for body-less HTTP/1.1 POST requests
 2016-02-23 15:00:33 -03:00
Elias Dorneles 329a77a3d0 Merge pull request #1808 from scrapy/revert-1701-cleanup-http11-tunneling 
Revert "Cleanup http11 tunneling connection after #1678"
 2016-02-23 13:59:15 -03:00
Elias Dorneles 10bcdb49b0 Merge pull request #1787 from scrapy/improve-errors 
[MRG+1] Better tracebacks
 2016-02-23 13:16:47 -03:00
Paul Tremberth 35fb630c4e Revert "[MRG+1] Cleanup http11 tunneling connection after #1678" 2016-02-23 16:28:48 +01:00
Mikhail Korobov 62a517163b Merge pull request #1801 from redapple/botocore-notconfigured-exception 
Fix SkipTest() message for botocore import test
 2016-02-21 12:33:48 +05:00
Paul Tremberth ad4c1169c4 Pass exception directly to SkipTest() 
It prints the same as passing `str(e)`
 2016-02-21 01:16:46 +01:00
Paul Tremberth e9bd328936 Fix SkipTest() message for botocore import test 2016-02-21 01:08:50 +01:00
Paul Tremberth ac8f97c33b Fix typo in comment 2016-02-21 00:49:41 +01:00
Paul Tremberth 6174192564 Add "Content-Length: 0" for body-less HTTP/1.1 POST requests 
GH-823 was fixed only for HTTP/1.0 (in GH-1089)
 2016-02-20 23:27:04 +01:00
Paul Tremberth 3f946b5278 Fix super() call 2016-02-20 02:07:45 +01:00
Paul Tremberth 49c757f208 Fix import for Ubuntu 12.04 precise (Twisted 11.1.0) 
Revert test server certificate change
 2016-02-20 01:58:59 +01:00
Paul Tremberth 57990fba7a Backward compatibility for HTTP/10 context factory 
New DOWNLOADER_CLIENT_TLS_METHOD setting to configure TLS method
 2016-02-20 01:32:21 +01:00
Paul Tremberth 2969e34e82 Merge pull request #1795 from AmbientLighter/patch-1 
Fix typos in docstrings
 2016-02-19 16:44:20 +01:00
Victor Mireyev 523e98dabe Fix typo in docstring 2016-02-19 17:32:01 +03:00
Victor Mireyev 5f5374209f Fix typo in docstring 2016-02-19 17:20:34 +03:00