RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,450 Commits 49 Branches 132 Tags 132 MiB
Commit Graph

6450 Commits

Author SHA1 Message Date
Paul Tremberth a68a8f8fdc Merge remote-tracking branch 'origin/1.4' 2017-08-05 11:45:50 +02:00
david watson 71d5b7d75a fix typo (#2867) 2017-08-01 14:49:22 -03:00
Andrei Petre 6e6b5cc29f Use getfullargspec under the scenes for py3 to stop DeprecationWarning (#2864) 
Use getfullargspec under the scenes for py3 to stop DeprecationWarning. 

Closes #2862
 2017-08-01 11:14:43 -03:00
Daniel Graña 19382c5675 Merge pull request #2755 from redapple/downloader-mdw-template 
[MRG+1] Add template for a downloader middleware
 2017-07-31 11:37:26 -03:00
simik-ru a65fec050a Small fix in description of startproject arguments 2017-07-31 13:03:49 +02:00
Paul Tremberth aaaa4da7a4 Add template for a downloader middleware 2017-07-28 13:48:23 +02:00
Mikhail Korobov 9d34b2fe16 Merge pull request #2857 from redapple/scrapy-components-logs-startup 
Log versions information at startup
 2017-07-28 00:02:06 +05:00
Paul Tremberth bf7ef3e4c3 Move methods to a new scrapy.utils.versions 2017-07-27 20:08:30 +02:00
Paul Tremberth 219c8aa0b6 Log versions information at startup 2017-07-27 17:30:30 +02:00
Daniel Graña 5d9bac789d Merge pull request #2849 from cclauss/patch-2 
[MRG+1] xrange() --> range() for Python 3
 2017-07-26 15:59:37 -03:00
Daniel Graña bbc56e6960 Merge pull request #2852 from starrify/new-http-codes-to-retry-2851 
Added: HTTP status code 522/524 to retry.
 2017-07-26 15:46:35 -03:00
Daniel Graña 881a5e3a57 Merge pull request #2847 from redapple/redirect-308 
Handle HTTP 308 Permanent Redirect
 2017-07-26 15:29:23 -03:00
Paul Tremberth 15a5c533fa Add tests for HTTP 307 permanent redirects 2017-07-26 19:07:57 +02:00
Paul Tremberth 1fdc10684f HTTP Cache: treat 308 as 301 2017-07-26 19:01:48 +02:00
Paul Tremberth 5dc9a88c34 Handle HTTP 308 Permanent Redirect 2017-07-26 19:01:48 +02:00
Pengyu Chen 11a1f970b7
Added: HTTP status code 522/524 to retry. 2017-07-26 16:11:13 +08:00
cclauss 33dfac5018 xrange() --> range() for Python 3 
Either this PR or #2845.
 2017-07-24 22:06:17 +02:00
Mikhail Korobov 17bbd71433 Merge pull request #2812 from elacuesta/inspect_response_populate_spider 
Populate spider variable when using shell.inspect_response
 2017-07-25 00:28:14 +05:00
Daniel Graña b6d036ede9 Merge pull request #2837 from dguo/patch-2 
Fix a typo in the Items documentation
 2017-07-24 15:55:30 -03:00
Daniel Graña 6de106e7a8 Merge pull request #2763 from scrapy/dataloss-typo 
DOC fixed rst syntax in DOWNLOAD_FAIL_ON_DATALOSS docs
 2017-07-24 15:48:03 -03:00
Daniel Graña 12971a7b61 Merge pull request #2764 from scrapy/readme-cleanup 
DOC change "releases" section content
 2017-07-24 15:47:39 -03:00
Daniel Graña c01c1ef2ae Merge pull request #2655 from Granitosaurus/ptpython_support 
[MRG+1] add support for embeded ptpython shell
 2017-07-24 15:42:00 -03:00
Daniel Graña 2371a2a0df Merge pull request #2789 from starrify/add-response-follow-tag-link 
[MRG+1] Added: Now supporting <link> tags in Response.follow
 2017-07-24 15:41:30 -03:00
Daniel Graña d1e948cef8 Merge pull request #2791 from starrify/doc-DontCloseSpider 
[MRG+1] Added doc for `scrapy.exceptions.DontCloseSpider`. Also fixes inaccurate doc for `scrapy.signals.spider_idle`.
 2017-07-24 15:29:38 -03:00
Daniel Graña 8e0b640d2b Merge pull request #2826 from dguo/patch-1 
[MRG+1] Tweak the CSVFeedSpider documentation
 2017-07-24 15:28:15 -03:00
Daniel Graña 45b81693b4 Merge pull request #2769 from stummjr/issue-2766 
[MRG+1] Add verification to check if Request callback is callable
 2017-07-24 15:14:33 -03:00
Daniel Graña 50b2567d0d Merge pull request #2848 from redapple/tox-jessie-ssl 
[WIP] Jessie toxenv: Add cryptography as per https://packages.debian.org/jessie/python-cryptography
 2017-07-24 14:44:41 -03:00
Paul Tremberth 1a18587d41 Jessi toxenv: Add cryptography as per https://packages.debian.org/jessie/python-cryptography 2017-07-24 19:30:08 +02:00
Valdir Stumm Junior bb0bd691d9 Improve error message when callback is not callable 2017-07-24 11:12:09 -03:00
Danny Guo 26c488970c Fix a typo in the Items documentation 2017-07-18 19:56:51 -05:00
Mikhail Korobov 17fe6d2553 Merge pull request #2828 from cconrad/patch-1 
Spelling mistake
 2017-07-15 12:09:45 +02:00
Claus Conrad 18b96dd82a Spelling mistake 2017-07-15 11:31:09 +02:00
Danny Guo dedc4a8b8f Tweak the CSVFeedSpider documentation 2017-07-13 22:58:10 -05:00
Mikhail Korobov 15abc0b9e3 Merge pull request #2816 from redapple/dns-cache-disabled 
Fix DNS resolver when DNSCACHE_ENABLED=False
 2017-07-06 15:53:33 +05:00
Paul Tremberth f0ded6b775 Do not cache DNS responses when cache size is 0 2017-07-04 23:18:15 +02:00
Paul Tremberth 1f08d9a648 Add test for DNS cache disabling 2017-07-04 23:10:19 +02:00
Eugenio Lacuesta 793b2376f8 Populate spider variable when using shell.inspect_response 2017-07-03 11:28:04 -03:00
Mikhail Korobov dd5ab6c5fe Merge pull request #2793 from lopuhin/pypy2 
Fix PyPy test failures
 2017-07-03 18:36:54 +05:00
Konstantin Lopuhin b0a9236357 Use environment markers for custom PyPy requirements 2017-06-19 19:16:50 +03:00
Konstantin Lopuhin 5ba8e5adc0 Remove duplicate PyPy toxenv from Travis config 
Thanks for the catch @redapple
 2017-06-19 17:45:28 +03:00
Konstantin Lopuhin 271b3a485c Require pypy build to pass 2017-06-19 16:46:16 +03:00
Konstantin Lopuhin ea08b95280 Remove Jython gc branch: it's not supported 2017-06-19 16:45:29 +03:00
Konstantin Lopuhin a8df090071 Fix httpcache leveldb tests: gc.collect after del 
LevelDB does not have "official" close method, so we have
to rely on garbage collection to close it.
 2017-06-15 14:29:28 +03:00
Konstantin Lopuhin b4eb60e527 Install PyPyDispatcher for PyPy tests 
Using https://github.com/lopuhin/pydispatcher, pypy branch.
This is executed as a separate step to avoid changing
default requirements.txt and setup.py. If just added to "deps"
in tox, this install command will be executed as one command
and PyPyDispatcher will not override PyDispatcher.
 2017-06-15 13:34:58 +03:00
Konstantin Lopuhin 19ca986aa1 Move garbage_collect to scrapy.utils.python 2017-06-15 13:14:31 +03:00
Konstantin Lopuhin 7c67047e77 Fix get_func_args tests under PyPy 
On CPython get_func_args does not work correctly for built-in
methods.
 2017-06-15 13:07:59 +03:00
Konstantin Lopuhin 5abb70c8d7 Fix test_weakkeycache on PyPy: run gc.collect() 
One gc.collect() seems to be enough, but it's more reliable
to run it several times (at most 100), until all objects are collected.
 2017-06-15 13:06:30 +03:00
Konstantin Lopuhin c3d17659b3 Fix queue serialization test on PyPy 
It is not affected by Twisted bug #7989 and is more permissive
with pickling (especially with protocol=2).
 2017-06-15 13:05:45 +03:00
Konstantin Lopuhin 6014856df5 Fix test_output_processor_error undere PyPy 
For float(u'$10') PyPy includes "u'" in the error message,
and it's more fair to check error message on input we are really
passing.
 2017-06-15 12:57:32 +03:00
Konstantin Lopuhin 5a08cf3b96 Fix test_start_requests_errors for PyPy 
Twisted prints errors in DebugInfo.__del__, but PyPy does not run
gc.collect() on exit:
http://doc.pypy.org/en/latest/cpython_differences.html?highlight=gc.collect#differences-related-to-garbage-collection-strategies
 2017-06-15 12:02:47 +03:00