RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,548 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

3691 Commits

Author SHA1 Message Date
Paul Tremberth ebbec3cd39 Catch VerificationError but keep the rest of ClientTLSOptions 2016-04-20 17:25:07 +02:00
Paul Tremberth 2f54f9680d Ignore HTTPS certificate verification failures 
Fixes #1930
 2016-04-20 17:24:49 +02:00
Paul Tremberth 7d3a529dbc Fix HTTP Pool key for HTTPS proxy tunneled connections (CONNECT method) 2016-04-20 17:02:47 +02:00
Paul Tremberth 352d0d36ee Fall back to no-SNi context factory is Twisted<14 is used 2016-04-20 16:48:32 +02:00
Paul Tremberth 5cb071b466 Set SNI properly when using CONNECT 2016-04-20 16:48:20 +02:00
Paul Tremberth f1020effc5 Merge pull request #1920 from scrapy/backport-1.1-pr1891 
[backport][1.1] Change Files/ImagesPipelines class attributes to instance attributes (PR #1891)
 2016-04-12 16:07:07 +02:00
djunzu 6d56faf11f Change ImagesPipeline class attributes to instance attributes. 
modified:   scrapy/pipelines/images.py
 2016-04-11 18:06:03 +02:00
djunzu bb68cc120b Change FilesPipeline class attributes to instance attributes. 
modified:   scrapy/pipelines/files.py
	modified:   tests/test_pipeline_files.py
 2016-04-11 18:06:03 +02:00
djunzu 6d8184953a Move default settings to settings/default_settings.py. 
modified:   scrapy/pipelines/files.py
	modified:   scrapy/pipelines/images.py
	modified:   scrapy/settings/default_settings.py
 2016-04-11 18:06:02 +02:00
nyov 1a62d87e35 Remove duplicate code now handled by newer w3lib 
see f3029a6a10
 2016-04-11 17:39:17 +02:00
Pengyu CHEN 160bcbacfc Added: Making it case-insensitive when extracting sitemap URLs from a robots.txt 2016-04-04 15:25:49 +02:00
Aron Bordin b2e914943d added FEED_TEMPDIR to settings 2016-04-04 13:59:04 +02:00
Konstantin Lopuhin b01bf4728f Fix a race condition in the FilesPipeline 
Checksum calculation could happen simultaniously with
persisting the file in the store (which is done in a thread):
they operated on the same buf object.
Concretely this lead to a bug with S3FilesStore
when using botocore: the signature did not match because
the position in the buf was already at the end.
The fix is to move checksum calculation before passing buf
to the store.
 2016-03-31 12:00:26 +02:00
Konstantin Lopuhin 84371ffb22 Fix handling of meta=None in S3FilesStore.persist_file 2016-03-31 12:00:26 +02:00
nyov 5ae8863d3d Rename isbinarytext function to binary_is_text for clarity 
Closes #1389
 2016-03-31 11:56:00 +02:00
pawelmhm b181c10fc4 response_status_message should not fail on non-standard HTTP codes 
utility is used in retry middleware and it was failing to handle non-standard HTTP codes.
Instead of raising exceptions when passing through to_native_str it should return
"Unknown status" message.
 2016-03-31 11:52:49 +02:00
djunzu 3d6abacc55 Add pps extension to IGNORED_EXTENSIONS 
modified:   scrapy/linkextractors/__init__.py
 2016-03-30 18:33:58 +02:00
nyov 1adc47ce93 Allow core Scheduler priority queue customization 2016-03-30 17:51:37 +02:00
nanolab e7a5e3717b Update httpcache.py 
It checks cache directory modification time, but have to check file modification time.
 2016-03-30 10:53:54 +02:00
Paul Tremberth e025f374e8 Bump version: 1.1.0rc2 → 1.1.0rc3 2016-03-01 23:40:30 +01:00
Paul Tremberth 4547a7e759 Explicitly call Twisted transport stopProducing() on HTTP/1.0 timeouts 2016-03-01 15:00:58 +01:00
Paul Tremberth a0c5c61855 Explicitly call Twisted transport stopProducing() on HTTP/1.1 timeouts 2016-03-01 15:00:44 +01:00
Mikhail Korobov a83ea5d50b Extract a function to build CONNECT request; add tests for it. See GH-1701 and GH-1808. 2016-03-01 14:38:39 +01:00
Paul Tremberth ab5e2accda Bump version: 1.1.0rc1 → 1.1.0rc2 2016-02-29 13:15:54 +01:00
Lucas Moauro 9766695550 Refactored SpiderLoader class constructor for easier subclassing 2016-02-27 14:47:20 +01:00
orangain 838bc08098 PY3: Fix TypeError when outputting to stdout 
This will fix #1768.
 2016-02-25 00:35:41 +01:00
Victor Mireyev bc170a2495 Fix typo in docstring 2016-02-25 00:28:54 +01:00
Paul Tremberth bdc1356871 Use best practices for TLS connections when using Twisted>=14.0 
Implement IPolicyForHTTPS
Backward compatibility for HTTP/10 context factory
New DOWNLOADER_CLIENT_TLS_METHOD setting to configure TLS method
Backward-compatibility for common Scrapy context factory patterns
Add warning for context factories not accepting `method` param
Use context factory class name in warning message
 2016-02-24 22:47:44 +01:00
Konstantin Lopuhin 9083b68125 Fix documentation for S3_STORE_ACL (now settings.FILES_STORE_S3_ACL) settings: it has nothing to do with feed exporters. 2016-02-24 22:26:22 +01:00
Konstantin Lopuhin b9c6d67365 See #1778 - change default S3 ACL to "private" and allow customization via settings 2016-02-24 22:26:22 +01:00
Paul Tremberth 6d4c51b266 Merge pull request #1809 from scrapy/backport-1.1-1787 
[backport][1.1] Better tracebacks
 2016-02-24 17:54:56 +01:00
Paul Tremberth f0c07a3121 Merge pull request #1810 from scrapy/backport-1.1-pr1800 
[backport][1.1] Add "Content-Length: 0" for body-less HTTP/1.1 POST requests
 2016-02-24 17:54:26 +01:00
Paul Tremberth 6801af2419 Merge pull request #1811 from scrapy/backport-1.1-revert-pr1678 
[backport][1.1] Revert "Cleanup http11 tunneling connection after #1678"
 2016-02-24 17:53:56 +01:00
Paul Tremberth 9d51e1fc3e Fix SkipTest() message for botocore import test 2016-02-24 17:42:42 +01:00
Paul Tremberth 0b2bb3bc7a Revert "[MRG+1] Cleanup http11 tunneling connection after #1678" 2016-02-24 17:24:54 +01:00
Paul Tremberth 2bcaff97b4 Fix typo in comment 2016-02-24 17:21:08 +01:00
Paul Tremberth e25052e4ef Add "Content-Length: 0" for body-less HTTP/1.1 POST requests 
GH-823 was fixed only for HTTP/1.0 (in GH-1089)
 2016-02-24 17:21:08 +01:00
Mikhail Korobov 374d7b04fc Remove unneeded `raise` (thanks @lopuhin). 2016-02-24 17:14:48 +01:00
Mikhail Korobov 0d596c27a9 Preserve tracebacks better. Fixes GH-1760. 2016-02-24 17:14:48 +01:00
Paul Tremberth a6bc758db1 Merge pull request #1792 from scrapy/backport-1.1-pr1761 
[backport][1.1.x] Py3 S3 botocore
 2016-02-19 11:17:20 +01:00
Paul Tremberth 3331cea788 More liberal Content-Disposition header parsing 
Fixes #1782
 2016-02-18 22:56:17 +01:00
Konstantin Lopuhin 25e481fd14 [backport][1.1.x] Py3 S3 botocore 
Originally from https://github.com/scrapy/scrapy/pull/1761
 2016-02-18 22:48:37 +01:00
Elias Dorneles f928014d91 add deprecation for pydispatch (thanks for the help @redapple) 2016-02-15 11:40:08 +01:00
Nicolas Pennequin 72229f92de MailSender.send: allow passing a charset. 
Resolves Issue #348
 2016-02-09 19:52:06 +01:00
Mikhail Korobov 8aca66252a more efficient ExecutionEngine.spider_is_idle 2016-02-09 19:52:06 +01:00
orangain 281e9bd046 PY3: Implement some attributes of WrappedRequest required in Python 3 
This will fix #1770.
 2016-02-09 16:10:58 +01:00
orangain 522c71bb8a Fix SitemapSpider to extract sitemap urls from robots.txt properly 
This will fix #1766.
 2016-02-09 16:10:58 +01:00
Elias Dorneles a0b63a7707 Bump version: 1.1.0dev1 → 1.1.0rc1 2016-02-03 21:52:47 -02:00
Daniel Graña a214b131fc Merge pull request #1748 from scrapy/py3-crawlerrunner-stop 
[MRG+1] PY3 fixed CrawlerRunner.stop
 2016-02-03 19:41:43 -03:00
Elias Dorneles a8a6f050e7 Merge pull request #1735 from ArturGaspar/master 
[MRG+1] Fix for KeyError in robots.txt middleware
 2016-02-03 13:30:06 -02:00