RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,404 Commits 49 Branches 132 Tags 138 MiB
Commit Graph

6404 Commits

Author SHA1 Message Date
Mikhail Korobov dd5ab6c5fe Merge pull request #2793 from lopuhin/pypy2 
Fix PyPy test failures
 2017-07-03 18:36:54 +05:00
Konstantin Lopuhin b0a9236357 Use environment markers for custom PyPy requirements 2017-06-19 19:16:50 +03:00
Konstantin Lopuhin 5ba8e5adc0 Remove duplicate PyPy toxenv from Travis config 
Thanks for the catch @redapple
 2017-06-19 17:45:28 +03:00
Konstantin Lopuhin 271b3a485c Require pypy build to pass 2017-06-19 16:46:16 +03:00
Konstantin Lopuhin ea08b95280 Remove Jython gc branch: it's not supported 2017-06-19 16:45:29 +03:00
Konstantin Lopuhin a8df090071 Fix httpcache leveldb tests: gc.collect after del 
LevelDB does not have "official" close method, so we have
to rely on garbage collection to close it.
 2017-06-15 14:29:28 +03:00
Konstantin Lopuhin b4eb60e527 Install PyPyDispatcher for PyPy tests 
Using https://github.com/lopuhin/pydispatcher, pypy branch.
This is executed as a separate step to avoid changing
default requirements.txt and setup.py. If just added to "deps"
in tox, this install command will be executed as one command
and PyPyDispatcher will not override PyDispatcher.
 2017-06-15 13:34:58 +03:00
Konstantin Lopuhin 19ca986aa1 Move garbage_collect to scrapy.utils.python 2017-06-15 13:14:31 +03:00
Konstantin Lopuhin 7c67047e77 Fix get_func_args tests under PyPy 
On CPython get_func_args does not work correctly for built-in
methods.
 2017-06-15 13:07:59 +03:00
Konstantin Lopuhin 5abb70c8d7 Fix test_weakkeycache on PyPy: run gc.collect() 
One gc.collect() seems to be enough, but it's more reliable
to run it several times (at most 100), until all objects are collected.
 2017-06-15 13:06:30 +03:00
Konstantin Lopuhin c3d17659b3 Fix queue serialization test on PyPy 
It is not affected by Twisted bug #7989 and is more permissive
with pickling (especially with protocol=2).
 2017-06-15 13:05:45 +03:00
Konstantin Lopuhin 6014856df5 Fix test_output_processor_error undere PyPy 
For float(u'$10') PyPy includes "u'" in the error message,
and it's more fair to check error message on input we are really
passing.
 2017-06-15 12:57:32 +03:00
Konstantin Lopuhin 5a08cf3b96 Fix test_start_requests_errors for PyPy 
Twisted prints errors in DebugInfo.__del__, but PyPy does not run
gc.collect() on exit:
http://doc.pypy.org/en/latest/cpython_differences.html?highlight=gc.collect#differences-related-to-garbage-collection-strategies
 2017-06-15 12:02:47 +03:00
Mikhail Korobov 5aebdac45d Merge pull request #2781 from crasker/patch-1 
use suggest method instead of DEPRECATED one
 2017-06-14 03:29:45 +05:00
Mikhail Korobov 4e84424f26 Merge pull request #2777 from chuanjin/patch-1 
Update extensions.rst
 2017-06-14 03:28:31 +05:00
Casker ae679f6499 Create item-pipeline.rst 2017-06-09 16:12:20 +08:00
Paul Tremberth 39ad0d0bdd Fix setting name reference 2017-06-06 10:48:30 +02:00
Paul Tremberth e7061f7a41 Reformat a bit 2017-06-06 10:47:43 +02:00
Chuan Jin 3f8542eb56 Update extensions.rst 
#2759
 2017-06-05 23:31:37 +02:00
Paul Tremberth fad6b70d92 Use https:// for readthedocs links 2017-06-01 16:41:52 +02:00
Paul Tremberth d847e65aae Merge pull request #2762 from stummjr/add-subreddit-to-docs 
Include references to Scrapy subreddit in the docs
 2017-05-29 20:35:40 +02:00
Valdir Stumm Junior 80b160d0d7 include references to scrapy subreddit in the docs 2017-05-29 14:56:49 -03:00
Kurt Peek af2963d0eb Update autothrottle.rst 
Added missing bullet point for the AUTOTHROTTLE_TARGET_CONCURRENCY setting.
 2017-05-24 19:59:36 +05:00
Paul Tremberth 5f69ec98f7 Bump version: 1.3.2 → 1.4.0 2017-05-18 23:01:05 +02:00
Paul Tremberth fc2846d637 Set release date for v1.4.0 2017-05-18 22:59:46 +02:00
Paul Tremberth 79b0d8605c Merge pull request #2630 from scrapy/release-notes-1.4.0 
Release notes for 1.4.0
 2017-05-18 22:57:51 +02:00
Mikhail Korobov 76e5b0f65c DOC 1.4 deprecations and backwards incompatible changes, add recent commits to news. 2017-05-18 22:37:15 +02:00
Mikhail Korobov edcde7a2cf DOC tweak release notes: promote response.follow, mention logging/stats changes 2017-05-18 22:37:15 +02:00
Paul Tremberth a3d3cd4cb7 Update with latest merges 2017-05-18 22:37:15 +02:00
Paul Tremberth 432668acf7 Mention implementation of #667 2017-05-18 22:37:15 +02:00
Paul Tremberth 896c30a8eb Reference items pretty-printing issue number 2017-05-18 22:37:15 +02:00
Paul Tremberth 55d1082360 Reference recent fixes and commits 2017-05-18 22:37:15 +02:00
Paul Tremberth 7d72394794 Reword mention of new response.follow() shortcut 2017-05-18 22:37:15 +02:00
Paul Tremberth c6464cc4f5 Add verbose introduction to new features 2017-05-18 22:37:14 +02:00
Paul Tremberth e139d990fc Fix sphinx-build warning on deprecated latex_paper_size 2017-05-18 22:37:14 +02:00
Paul Tremberth cba55cd190 Rephrase other sections 2017-05-18 22:37:14 +02:00
Paul Tremberth 8729a91f7a Rephrase "New features" section 2017-05-18 22:37:14 +02:00
Paul Tremberth 851adcedf2 List merged pull requests since 1.3.3 2017-05-18 22:37:14 +02:00
Paul Tremberth 0c237c6cac Merge pull request #2750 from scrapy/codecov 
tweak coverage reports
 2017-05-18 21:38:54 +02:00
Daniel Graña 25f609e2a3 Merge pull request #2675 from simongartz/png-p-to-jpg-conversion-fix 
[MRG+1] Fixes conversion of transparent PNG with palette images to jpg #2452
 2017-05-18 16:34:02 -03:00
Paul Tremberth 8aa2e4f997 Merge pull request #1829 from nyov/nyov/editor 
[MRG+1] Remove dependency on os.environ from default settings
 2017-05-18 21:32:51 +02:00
Daniel Graña b9d3b447a5 Merge pull request #2670 from qhuang872/master 
[MRG+1] Update spiders.rst
 2017-05-18 16:31:01 -03:00
Paul Tremberth 143810410f Merge pull request #2740 from luzfcb/luzfcb-run-scrapy-from-module 
[MRG+1] Add support for executing scrapy using -m option of python
 2017-05-18 21:30:19 +02:00
Mikhail Korobov 9ce03d096d codecov config: disable project check, tweak PR comments 2017-05-19 00:01:27 +05:00
Mikhail Korobov 532400f993 Merge pull request #2643 from harshasrinivas/set-retry-times-per-request 
[MRG+1] Add feature to set RETRY_TIMES per request (#2642)
 2017-05-17 15:46:33 +05:00
Mikhail Korobov 1acacab93b Merge pull request #2741 from eliat123/2576_cleanup_MEMUSAGE_REPORT 
[FIX #2576] cleanup: removed unused MEMUSAGE_REPORT
 2017-05-17 13:48:00 +05:00
Daniel Graña 73668ce407 Merge pull request #2721 from HarrisonGregg/feature-drop-from-response-field 
[MRG+1] Allow dropping field in from_response formdata
 2017-05-16 09:47:45 -03:00
Eli Atzaba b74b98fa3e cleanup: removed unused MEMUSAGE_REPORT 
Signed-off-by: Eli Atzaba <eliat123@gmail.com>
 2017-05-16 13:59:58 +03:00
Fábio C. Barrionuevo da Luz df7a5c4aa4 Add support for executing scrapy using -m option of python 
python -m scrapy
 2017-05-15 22:52:23 -03:00
Harrison Gregg ffef828a8d Add test for dropping fields in from_response request body 2017-05-15 12:25:32 -04:00