RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,679 Commits 49 Branches 132 Tags 138 MiB
Commit Graph

6679 Commits

Author SHA1 Message Date
Konstantin Lopuhin b3cd12dc48 Try to get python3.7 by using xenial base and sudo 
See https://github.com/travis-ci/travis-ci/issues/9815#issuecomment-401756442
and https://github.com/travis-ci/travis-ci/issues/9815#issuecomment-402045581
 2018-07-09 13:12:47 +03:00
Konstantin Lopuhin f4f39057cb Make csviter work on python 3.7 
PEP 479 does not allow for StopIteration in generators.  Instead,
handle it explicitly, also use a for loop which looks simpler.
 2018-07-09 12:46:45 +03:00
Konstantin Lopuhin 2773fe09e4 Make "docs" the last build, even though it still uses python3.6 for now 2018-07-09 12:36:58 +03:00
Konstantin Lopuhin cf9399acc1 Use python 3.7 on travis 2018-07-09 12:26:56 +03:00
Konstantin Lopuhin 17e9914b8a Catch SyntaxError as well when importing manhole 
Also give a more detailed reason why telnet is not enabled (for the
future).
 2018-07-09 12:26:09 +03:00
Konstantin Lopuhin 722e1afcdb Update ancient pytest on python 3 
2.9 gives collection errors on python 3.7 due to PEP 479.
 2018-07-09 12:21:19 +03:00
Konstantin Lopuhin 1d2e2735fc Merge branch 'master' of https://github.com/patiences/scrapy into patiences-master 2018-07-09 11:59:22 +03:00
Daniel Graña 666e23714a
Merge pull request #3318 from scrapy/delete-relocation-shims 
[MRG+1] Delete relocation shims
 2018-07-06 13:49:59 -03:00
Mikhail Korobov f531b66822 SpiderManager shim is removed 2018-07-06 03:28:01 +05:00
Mikhail Korobov 36453348fa remove ancient modules kept only for error messages 2018-07-06 03:23:37 +05:00
Mikhail Korobov d4c7cc848b remove backwards compatibility shims for relocated modules 2018-07-06 03:19:43 +05:00
Oz T 6f5c39d65f Fix for CSV export unnecessary blank lines problem on Windows (#3039) 2018-07-03 18:22:24 -03:00
chainly 74ce156154 add item_error to be catchable (#3256) 2018-07-03 16:00:59 -03:00
Grammy Jiang d05c8677c5 [MRG+1] change the bad smell code (#3304) 
Change the bad smell code
 2018-07-03 15:58:43 -03:00
Daniel Graña 8d93691a8a
Merge pull request #3311 from scrapy/rename-requirements 
[MRG+1] TST make it clear which requirements are Python 2-only
 2018-07-03 10:33:26 -03:00
Mikhail Korobov f11d65f7d6 TST make it clear which requirements are Python 2-only 
* rename requirements.txt to requirements-py2.txt, to make it clear they are Python 2-only
* make requirements-py3.txt consistent with requirements-py2.txt
 2018-06-29 18:34:11 +05:00
Daniel Graña 64f48efa26
Merge pull request #3308 from scrapy/fix-build 
TST exclude lxml==4.2.2 from tests
 2018-06-27 23:42:32 -03:00
Mikhail Korobov 8782901fc8
[MRG+1] TST test agains latest pypy (#3309) 
pypy3 is not upgraded, as tests segfault with pypy3 6.0 for some reason
 2018-06-28 01:11:15 +05:00
Mikhail Korobov 45f67eb64d TST exclude lxml==4.2.2 from tests, as it doesn't play well with Pillow 2018-06-28 01:02:30 +05:00
Mikhail Korobov fac1b2f351 TST remove workaround for old Pillow versions which don't support BytesIO 2018-06-28 01:01:27 +05:00
Daniel Graña bc4dbd2766
Merge pull request #3305 from grammy-jiang/patch-1 
Make the version of ipython less than 6.0 in python 2.7
 2018-06-27 11:19:32 -03:00
Daniel Graña 594a441eb5
Merge pull request #3294 from rennerocha/feed-exporter-docs 
[MRG+1] Improve docs of S3 Storage to make Python version more explicit
 2018-06-27 11:16:07 -03:00
Grammy Jiang 9ad3af9d88
Update requirements.txt 
make the version of ipython less than 6.0 in python 2.7
 2018-06-23 17:31:54 +08:00
Mikhail Korobov 1fd1702a11
Merge pull request #3299 from grammy-jiang/dev 
[MRG+1] fix the test case name of HttpProxyMiddleware
 2018-06-22 04:17:18 +05:00
Grammy Jiang 88bd067912 fix the test case name of HttpProxyMiddleware 2018-06-20 16:56:46 +08:00
Konstantin Lopuhin 991e9b889f
Merge pull request #3298 from leo8a/master 
fix typo in news.rst: extractred --> extracted
 2018-06-19 18:10:14 +03:00
Leo 7a601d76de fix typo 
extractred --> extracted
 2018-06-19 10:51:55 +02:00
Vostretsov Nikita 72d0899bce Return non-zero exit code from scrapy commands in case of spider bootstrap errors 
* method to detect spider creation in crawler

* correct method name

* method to know if crawlers has spiders

* we do not need to issue requests

* set exit code accordingly to spiders in crawlers

* more portable way to check ofr exceptions

* more clear way

* test cases for several spiders per crawler

* grammatically correct name for method

* method is private

* grammatically correct name for method

* method is private

* remove unused import

* correct order of imports

* changes mechanism of obtaining spider status from method to object member

* rename tests
 2018-06-14 19:58:48 +05:00
Mikhail Korobov 667eb7151b
Merge pull request #3284 from mugayoshi/issue3282 
[MRG+1] Update debugging memory leaks section in the docs
 2018-06-14 19:56:52 +05:00
Renne Rocha e2bb218e9a Include Python version indication to each required library used in S3 storage 2018-06-13 18:11:43 -03:00
mugayoshi d4511667fb Update debugging memory leaks section in the docs 
Add Python3 tools description.
 2018-06-09 18:17:11 +09:00
Daniel Graña c6030ce8c6
Merge pull request #3231 from starrify/updating-argument-of-cookiejar-clear 
[MRG+2] Added: Allowing optional arguments for `scrapy.http.cookies.CookieJar.clear`
 2018-06-01 21:52:32 -03:00
Colton Herinckx 596f39600d reversed earlier change that seemed to cause Travis CI build failure 2018-06-01 21:48:43 -03:00
Colton Herinckx 12d10eec2c changed Twisted >= 17.9.0 to Twisted>=17.9.0 2018-06-01 21:48:43 -03:00
Colton Herinckx 9bd5444a42 added oxford commas to LICENSE 2018-06-01 21:48:43 -03:00
Colton Herinckx 98d9093dc7 minor grammatical fixes in CODE_OF_CONDUCT.md 2018-06-01 21:48:43 -03:00
Mikhail Korobov 3cf871c61f
Merge pull request #3281 from fbergen/gunzipperf 
[MRG+2] Improve gunzip performance for big files on Python 3
 2018-06-02 04:13:26 +05:00
Mikhail Korobov 847b50ce9c
Merge pull request #3201 from grammy-jiang/master 
[MRG+1] fix a mistake in topic spider-middleware.rst
 2018-06-02 01:09:59 +05:00
Fredrik Bergenlid 6a2d2c3b77 Improve gunzip performance for big files 2018-06-01 21:38:07 +02:00
Mikhail Korobov 13b15dc9a8
Merge pull request #3279 from lewoudar/patch-1 
Update spiders.rst
 2018-06-01 22:20:41 +05:00
Mikhail Korobov f1d87ee09a
Merge pull request #3280 from cms-/patch-1 
Minor edits to contributing.rst
 2018-06-01 20:54:39 +05:00
Chris Slothouber ecdd888ff4
Minor edits to contributing.rst 
Corrected minor grammatical issues and increased clarity of instructions.
 2018-06-01 09:25:34 -04:00
Kevin Tewouda ffa7bede17
Update spiders.rst 
I changed URLs to :class:`~scrapy.http.Request` in start_urls explanation of the default spider
 2018-05-30 06:33:18 +02:00
Vostretsov Nikita b364d27247 [MRG+1] Automatic port selection for servicies in unit tests (#3210) 
* ability to pass port as a parameter

* try to find free ports

* use environment variables to pass mock server address

* get mock server address from environment variables

* ability to select ports for proxy in runtime

* use common method for URLs from mock server

* https support

* get mock server address

* get mock address

* replace hand-written mechanism by kernel-based one

* use ephemeral ports in mockserver

* strip EOL from addresses

* use ephemeral port in proxy

* no need to restore environment as it is restored in tearDown

* decode bytes

* use mockserver address as a variable

* ability to pass address as variable

* per test-case mockserver

* use base class

* remove obsolete environment manipulation

* return usage of proxy for http cases

* common method for broking proxy auth credentials

* python version-independent url methods
 2018-05-23 21:25:50 +03:00
Konstantin Lopuhin bac1e2d47d
Merge pull request #3263 from whalebot-helmsman/no_twisted_18_4_0 
[MRG+1] Blacklist twisted version with regression
 2018-05-17 13:21:42 +03:00
Vostretsov Nikita c5ddfddb7e blacklist twisted version with regression in constraints file 2018-05-17 08:53:42 +00:00
Vostretsov Nikita 0d015e5c0f blacklist twisted version with regression 2018-05-16 09:36:07 +00:00
Konstantin Lopuhin c4f096d3a5
Merge pull request #3224 from lucywang000/better-processors-doc 
[MRG+1] Improve document about functions as processors
 2018-04-26 09:36:39 +03:00
Konstantin Lopuhin f36e1b53d4
Merge pull request #3199 from rhoboro/gcs_acl 
[MRG+1] FilesPipeline supports ACL for Google Cloud Storage
 2018-04-25 19:42:19 +03:00
Pengyu Chen e75f721c04
Added: Allowing optional arguments for `scrapy.http.cookies.CookieJar.clear` 2018-04-23 22:08:28 +08:00