RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,570 Commits 49 Branches 132 Tags 138 MiB
Commit Graph

5570 Commits

Author SHA1 Message Date
Paul Tremberth 9ae4e46f32 Merge pull request #1883 from lopuhin/botocore-files-store-fix 
[MRG+1] Make FilesPipeline work with S3FilesStore using botocore
 2016-03-31 11:57:39 +02:00
Paul Tremberth 3ba5671fbc Merge pull request #1851 from nyov/binary_or_text 
[MRG+1] Rename isbinarytext function to binary_is_text for clarity
 2016-03-31 11:55:09 +02:00
Paul Tremberth 3a763f7ba7 Merge pull request #1857 from pawelmhm/fix_response_status_msg 
[MRG+1] response_status_message should not fail on non-standard HTTP codes
 2016-03-31 11:44:44 +02:00
nyov e8ca467572 Rename isbinarytext function to binary_is_text for clarity 
Closes #1389
 2016-03-30 15:44:15 +00:00
Mikhail Korobov a38a99e0e2 Merge pull request #1893 from redapple/sphinx-1.4 
Add support for Sphinx 1.4
 2016-03-30 19:55:48 +06:00
Paul Tremberth 1075587dbd Add support for Sphinx 1.4 
See http://www.sphinx-doc.org/en/stable/changes.html#release-1-4-released-mar-28-2016

sphinx_rtd_theme has become optional, needs to be added to reqs

https://github.com/sphinx-doc/sphinx/pull/2320 changes node entries tuples
to 5 values instead of 4

`sh` syntax highlighting added very locally in selectors.rst
because of this warning/error with Sphinx 1.4:

```
Warning, treated as error:
/home/paul/src/scrapy/docs/topics/selectors.rst:743:
WARNING: Could not lex literal_block as "python". Highlighting skipped.
```
 2016-03-30 14:40:52 +02:00
nanolab a583e4d531 Update httpcache.py 
It checks cache directory modification time, but have to check file modification time.
 2016-03-30 10:57:48 +02:00
Lele 7082454f2a Changed sel. to response. for clarity 
Changed sel. to response. to comply with the rest of the examples in the same section, to avoid confusion.
 2016-03-28 05:27:15 +05:00
Konstantin Lopuhin fc8cd45a48 Fix a race condition in the FilesPipeline 
Checksum calculation could happen simultaniously with
persisting the file in the store (which is done in a thread):
they operated on the same buf object.
Concretely this lead to a bug with S3FilesStore
when using botocore: the signature did not match because
the position in the buf was already at the end.
The fix is to move checksum calculation before passing buf
to the store.
 2016-03-27 21:56:47 +02:00
Konstantin Lopuhin 5045a4f168 Fix handling of meta=None in S3FilesStore.persist_file 2016-03-25 18:35:55 +03:00
pawelmhm 65c7c05060 response_status_message should not fail on non-standard HTTP codes 
utility is used in retry middleware and it was failing to handle non-standard HTTP codes.
Instead of raising exceptions when passing through to_native_str it should return
"Unknown status" message.
 2016-03-12 14:16:40 +01:00
Mikhail Korobov ebef6d7c6d Merge pull request #1848 from aron-bordin/small_doc_style_fixes 
small doc style fixes
 2016-03-07 08:49:25 +05:00
Aron Bordin 2cfe9e424d small doc style fixes 2016-03-05 19:54:06 -03:00
Paul Tremberth e122c569fe Merge pull request #1842 from nyov/nyov/docs 
[MRG+1] Update documentation links
 2016-03-04 11:50:26 +01:00
nyov 5876b9aa30 Update documentation links 2016-03-03 16:28:33 +00:00
Paul Tremberth 9f4fe5dc4a Merge pull request #1822 from nyov/nyov/scheduler 
[MRG+1] Allow core Scheduler priority queue customization
 2016-03-02 14:20:40 +01:00
Mikhail Korobov 6b2871dadd Merge pull request #1835 from djunzu/add_pps_to_IGNORED_EXTENSIONS 
[MRG+1] Add pps extension to IGNORED_EXTENSIONS
 2016-03-02 16:15:51 +05:00
djunzu 0e288d4a71 Add pps extension to IGNORED_EXTENSIONS 
modified:   scrapy/linkextractors/__init__.py
 2016-03-01 21:02:13 -03:00
nyov 2a6524ee3a Allow core Scheduler priority queue customization 2016-03-01 13:58:40 +00:00
Daniel Graña b8fcb46e67 Merge pull request #1804 from redapple/enable-test-dwnld-timeout 
Re-enable HTTPS tests for download timeouts
 2016-03-01 10:44:24 -03:00
Daniel Graña 21da493109 Merge pull request #1828 from scrapy/py3-classifiers 
[MRG+1] declare Python 3 support in setup.py
 2016-03-01 10:34:36 -03:00
Daniel Graña cf535fe840 Merge pull request #1827 from scrapy/proxy-auth-test 
[MRG+1] Extract a function to build CONNECT request; add tests for it.
 2016-03-01 10:34:24 -03:00
Mikhail Korobov 17d3bec699 declare Python 3 support in setup.py 2016-03-01 16:34:13 +05:00
Mikhail Korobov 94e28adfb7 Extract a function to build CONNECT request; add tests for it. See GH-1701 and GH-1808. 2016-03-01 16:29:12 +05:00
Mikhail Korobov e8635cd03c Merge pull request #1826 from redapple/universal-wheels 
Build universal wheels
 2016-03-01 15:23:57 +05:00
Paul Tremberth f3c7a509b9 Build universal wheels 2016-03-01 11:00:20 +01:00
Mikhail Korobov 16512a32ea Merge pull request #1825 from redapple/relnotes-s3policy 
[MRG+1] Update release notes about change of default S3 ACL policy to "private"
 2016-02-29 17:02:18 +05:00
Paul Tremberth 90c64a6d13 Update release notes about change of default S3 ACL policy to "private" 2016-02-29 12:27:25 +01:00
Paul Tremberth 74158611c8 Merge pull request #1818 from lagenar/master 
[MRG+1] Refactored SpiderLoader class constructor for easier subclassing
 2016-02-26 23:23:34 +01:00
Paul Tremberth 241ae9f22f Merge pull request #1820 from redapple/http-tls-settings 
[MRG+1] Document DOWNLOADER_* settings for HTTP/1.0 and TLS
 2016-02-26 23:20:33 +01:00
Paul Tremberth 709b4fa884 Update release notes about HTTPS downloader 2016-02-26 18:35:29 +01:00
Paul Tremberth 174f526709 Document DOWNLOADER_* settings for HTTP/1.0 and TLS 2016-02-26 18:23:57 +01:00
Lucas Moauro 1ce4c86c2b Refactored SpiderLoader class constructor for easier subclassing 2016-02-25 21:08:50 -03:00
Paul Tremberth 84dea19455 Update release notes 2016-02-25 00:59:15 +01:00
Paul Tremberth c9e781357d Explicitly call Twisted transport stopProducing() on HTTP/1.0 timeouts 2016-02-24 23:04:31 +01:00
Paul Tremberth ecddc093a4 Explicitly call Twisted transport stopProducing() on HTTP/1.1 timeouts 2016-02-24 23:04:31 +01:00
Mikhail Korobov f7a48b0c6b Merge pull request #1794 from redapple/twisted-tls 
[MRG+1] Use best practices for TLS connections when using Twisted>=14.0
 2016-02-25 02:23:33 +05:00
Paul Tremberth 4e93501304 Merge pull request #1796 from lopuhin/s3-acl-private 
[MRG+1] Change default S3 ACL to "private" and allow customization via settings
 2016-02-24 22:18:43 +01:00
Paul Tremberth 0336c2504a Use context factory class name in warning message 2016-02-24 16:42:25 +01:00
Paul Tremberth c29a1b98b3 Make warning message a bit nicer in logs 2016-02-24 16:01:18 +01:00
Konstantin Lopuhin 6137dd96d9 Fix documentation for S3_STORE_ACL (now settings.FILES_STORE_S3_ACL) settings: it has nothing to do with feed exporters. 2016-02-24 10:16:41 +03:00
Konstantin Lopuhin 164f300762 See #1778 - change default S3 ACL to "private" and allow customization via settings 2016-02-24 10:16:41 +03:00
Paul Tremberth c9890d5f76 Add warning for context factories not accepting `method` param 2016-02-24 01:26:04 +01:00
Paul Tremberth 095495e99c Backward-compatibility for common Scrapy context factory patterns 2016-02-24 01:24:58 +01:00
Daniel Graña 513ba7a1fb Merge pull request #1800 from redapple/http11-post-content-length 
[MRG+1] Add "Content-Length: 0" for body-less HTTP/1.1 POST requests
 2016-02-23 15:00:33 -03:00
Elias Dorneles 329a77a3d0 Merge pull request #1808 from scrapy/revert-1701-cleanup-http11-tunneling 
Revert "Cleanup http11 tunneling connection after #1678"
 2016-02-23 13:59:15 -03:00
Elias Dorneles 10bcdb49b0 Merge pull request #1787 from scrapy/improve-errors 
[MRG+1] Better tracebacks
 2016-02-23 13:16:47 -03:00
Paul Tremberth 35fb630c4e Revert "[MRG+1] Cleanup http11 tunneling connection after #1678" 2016-02-23 16:28:48 +01:00
Mikhail Korobov 62a517163b Merge pull request #1801 from redapple/botocore-notconfigured-exception 
Fix SkipTest() message for botocore import test
 2016-02-21 12:33:48 +05:00
Paul Tremberth ad4c1169c4 Pass exception directly to SkipTest() 
It prints the same as passing `str(e)`
 2016-02-21 01:16:46 +01:00