92b504eae5
Fix telnet warnings in tests
...
Disable telnet console if it's not available, else we'll get an extra
warning about failure to enable it, and tests will fail.
2018-07-09 13:44:02 +03:00
b3cd12dc48
Try to get python3.7 by using xenial base and sudo
...
See https://github.com/travis-ci/travis-ci/issues/9815#issuecomment-401756442
and https://github.com/travis-ci/travis-ci/issues/9815#issuecomment-402045581
2018-07-09 13:12:47 +03:00
f4f39057cb
Make csviter work on python 3.7
...
PEP 479 does not allow for StopIteration in generators. Instead,
handle it explicitly, also use a for loop which looks simpler.
2018-07-09 12:46:45 +03:00
2773fe09e4
Make "docs" the last build, even though it still uses python3.6 for now
2018-07-09 12:36:58 +03:00
cf9399acc1
Use python 3.7 on travis
2018-07-09 12:26:56 +03:00
17e9914b8a
Catch SyntaxError as well when importing manhole
...
Also give a more detailed reason why telnet is not enabled (for the
future).
2018-07-09 12:26:09 +03:00
722e1afcdb
Update ancient pytest on python 3
...
2.9 gives collection errors on python 3.7 due to PEP 479.
2018-07-09 12:21:19 +03:00
1d2e2735fc
Merge branch 'master' of https://github.com/patiences/scrapy into patiences-master
2018-07-09 11:59:22 +03:00
666e23714a
Merge pull request #3318 from scrapy/delete-relocation-shims
...
[MRG+1] Delete relocation shims
2018-07-06 13:49:59 -03:00
f531b66822
SpiderManager shim is removed
2018-07-06 03:28:01 +05:00
36453348fa
remove ancient modules kept only for error messages
2018-07-06 03:23:37 +05:00
d4c7cc848b
remove backwards compatibility shims for relocated modules
2018-07-06 03:19:43 +05:00
6f5c39d65f
Fix for CSV export unnecessary blank lines problem on Windows ( #3039 )
2018-07-03 18:22:24 -03:00
74ce156154
add item_error to be catchable ( #3256 )
2018-07-03 16:00:59 -03:00
d05c8677c5
[MRG+1] change the bad smell code ( #3304 )
...
Change the bad smell code
2018-07-03 15:58:43 -03:00
8d93691a8a
Merge pull request #3311 from scrapy/rename-requirements
...
[MRG+1] TST make it clear which requirements are Python 2-only
2018-07-03 10:33:26 -03:00
f11d65f7d6
TST make it clear which requirements are Python 2-only
...
* rename requirements.txt to requirements-py2.txt, to make it clear they are Python 2-only
* make requirements-py3.txt consistent with requirements-py2.txt
2018-06-29 18:34:11 +05:00
64f48efa26
Merge pull request #3308 from scrapy/fix-build
...
TST exclude lxml==4.2.2 from tests
2018-06-27 23:42:32 -03:00
8782901fc8
[MRG+1] TST test agains latest pypy ( #3309 )
...
pypy3 is not upgraded, as tests segfault with pypy3 6.0 for some reason
2018-06-28 01:11:15 +05:00
45f67eb64d
TST exclude lxml==4.2.2 from tests, as it doesn't play well with Pillow
2018-06-28 01:02:30 +05:00
fac1b2f351
TST remove workaround for old Pillow versions which don't support BytesIO
2018-06-28 01:01:27 +05:00
bc4dbd2766
Merge pull request #3305 from grammy-jiang/patch-1
...
Make the version of ipython less than 6.0 in python 2.7
2018-06-27 11:19:32 -03:00
594a441eb5
Merge pull request #3294 from rennerocha/feed-exporter-docs
...
[MRG+1] Improve docs of S3 Storage to make Python version more explicit
2018-06-27 11:16:07 -03:00
9ad3af9d88
Update requirements.txt
...
make the version of ipython less than 6.0 in python 2.7
2018-06-23 17:31:54 +08:00
1fd1702a11
Merge pull request #3299 from grammy-jiang/dev
...
[MRG+1] fix the test case name of HttpProxyMiddleware
2018-06-22 04:17:18 +05:00
88bd067912
fix the test case name of HttpProxyMiddleware
2018-06-20 16:56:46 +08:00
991e9b889f
Merge pull request #3298 from leo8a/master
...
fix typo in news.rst: extractred --> extracted
2018-06-19 18:10:14 +03:00
7a601d76de
fix typo
...
extractred --> extracted
2018-06-19 10:51:55 +02:00
72d0899bce
Return non-zero exit code from scrapy commands in case of spider bootstrap errors
...
* method to detect spider creation in crawler
* correct method name
* method to know if crawlers has spiders
* we do not need to issue requests
* set exit code accordingly to spiders in crawlers
* more portable way to check ofr exceptions
* more clear way
* test cases for several spiders per crawler
* grammatically correct name for method
* method is private
* grammatically correct name for method
* method is private
* remove unused import
* correct order of imports
* changes mechanism of obtaining spider status from method to object member
* rename tests
2018-06-14 19:58:48 +05:00
667eb7151b
Merge pull request #3284 from mugayoshi/issue3282
...
[MRG+1] Update debugging memory leaks section in the docs
2018-06-14 19:56:52 +05:00
e2bb218e9a
Include Python version indication to each required library used in S3 storage
2018-06-13 18:11:43 -03:00
d4511667fb
Update debugging memory leaks section in the docs
...
Add Python3 tools description.
2018-06-09 18:17:11 +09:00
c6030ce8c6
Merge pull request #3231 from starrify/updating-argument-of-cookiejar-clear
...
[MRG+2] Added: Allowing optional arguments for `scrapy.http.cookies.CookieJar.clear`
2018-06-01 21:52:32 -03:00
596f39600d
reversed earlier change that seemed to cause Travis CI build failure
2018-06-01 21:48:43 -03:00
12d10eec2c
changed Twisted >= 17.9.0 to Twisted>=17.9.0
2018-06-01 21:48:43 -03:00
9bd5444a42
added oxford commas to LICENSE
2018-06-01 21:48:43 -03:00
98d9093dc7
minor grammatical fixes in CODE_OF_CONDUCT.md
2018-06-01 21:48:43 -03:00
3cf871c61f
Merge pull request #3281 from fbergen/gunzipperf
...
[MRG+2] Improve gunzip performance for big files on Python 3
2018-06-02 04:13:26 +05:00
847b50ce9c
Merge pull request #3201 from grammy-jiang/master
...
[MRG+1] fix a mistake in topic spider-middleware.rst
2018-06-02 01:09:59 +05:00
6a2d2c3b77
Improve gunzip performance for big files
2018-06-01 21:38:07 +02:00
13b15dc9a8
Merge pull request #3279 from lewoudar/patch-1
...
Update spiders.rst
2018-06-01 22:20:41 +05:00
f1d87ee09a
Merge pull request #3280 from cms-/patch-1
...
Minor edits to contributing.rst
2018-06-01 20:54:39 +05:00
ecdd888ff4
Minor edits to contributing.rst
...
Corrected minor grammatical issues and increased clarity of instructions.
2018-06-01 09:25:34 -04:00
ffa7bede17
Update spiders.rst
...
I changed URLs to :class:`~scrapy.http.Request` in start_urls explanation of the default spider
2018-05-30 06:33:18 +02:00
b364d27247
[MRG+1] Automatic port selection for servicies in unit tests ( #3210 )
...
* ability to pass port as a parameter
* try to find free ports
* use environment variables to pass mock server address
* get mock server address from environment variables
* ability to select ports for proxy in runtime
* use common method for URLs from mock server
* https support
* get mock server address
* get mock address
* replace hand-written mechanism by kernel-based one
* use ephemeral ports in mockserver
* strip EOL from addresses
* use ephemeral port in proxy
* no need to restore environment as it is restored in tearDown
* decode bytes
* use mockserver address as a variable
* ability to pass address as variable
* per test-case mockserver
* use base class
* remove obsolete environment manipulation
* return usage of proxy for http cases
* common method for broking proxy auth credentials
* python version-independent url methods
2018-05-23 21:25:50 +03:00
bac1e2d47d
Merge pull request #3263 from whalebot-helmsman/no_twisted_18_4_0
...
[MRG+1] Blacklist twisted version with regression
2018-05-17 13:21:42 +03:00
c5ddfddb7e
blacklist twisted version with regression in constraints file
2018-05-17 08:53:42 +00:00
0d015e5c0f
blacklist twisted version with regression
2018-05-16 09:36:07 +00:00
c4f096d3a5
Merge pull request #3224 from lucywang000/better-processors-doc
...
[MRG+1] Improve document about functions as processors
2018-04-26 09:36:39 +03:00
f36e1b53d4
Merge pull request #3199 from rhoboro/gcs_acl
...
[MRG+1] FilesPipeline supports ACL for Google Cloud Storage
2018-04-25 19:42:19 +03:00
Konstantin Lopuhin
Daniel Graña
Mikhail Korobov
Oz T
chainly
Grammy Jiang
Grammy Jiang
Leo
Vostretsov Nikita
Renne Rocha
mugayoshi
Colton Herinckx
Fredrik Bergenlid
Chris Slothouber
Kevin Tewouda