RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,652 Commits 49 Branches 132 Tags 149 MiB
Commit Graph

6652 Commits

Author SHA1 Message Date
Mikhail Korobov 45f67eb64d TST exclude lxml==4.2.2 from tests, as it doesn't play well with Pillow 2018-06-28 01:02:30 +05:00
Mikhail Korobov fac1b2f351 TST remove workaround for old Pillow versions which don't support BytesIO 2018-06-28 01:01:27 +05:00
Mikhail Korobov 1fd1702a11
Merge pull request #3299 from grammy-jiang/dev 
[MRG+1] fix the test case name of HttpProxyMiddleware
 2018-06-22 04:17:18 +05:00
Grammy Jiang 88bd067912 fix the test case name of HttpProxyMiddleware 2018-06-20 16:56:46 +08:00
Konstantin Lopuhin 991e9b889f
Merge pull request #3298 from leo8a/master 
fix typo in news.rst: extractred --> extracted
 2018-06-19 18:10:14 +03:00
Leo 7a601d76de fix typo 
extractred --> extracted
 2018-06-19 10:51:55 +02:00
Vostretsov Nikita 72d0899bce Return non-zero exit code from scrapy commands in case of spider bootstrap errors 
* method to detect spider creation in crawler

* correct method name

* method to know if crawlers has spiders

* we do not need to issue requests

* set exit code accordingly to spiders in crawlers

* more portable way to check ofr exceptions

* more clear way

* test cases for several spiders per crawler

* grammatically correct name for method

* method is private

* grammatically correct name for method

* method is private

* remove unused import

* correct order of imports

* changes mechanism of obtaining spider status from method to object member

* rename tests
 2018-06-14 19:58:48 +05:00
Mikhail Korobov 667eb7151b
Merge pull request #3284 from mugayoshi/issue3282 
[MRG+1] Update debugging memory leaks section in the docs
 2018-06-14 19:56:52 +05:00
mugayoshi d4511667fb Update debugging memory leaks section in the docs 
Add Python3 tools description.
 2018-06-09 18:17:11 +09:00
Daniel Graña c6030ce8c6
Merge pull request #3231 from starrify/updating-argument-of-cookiejar-clear 
[MRG+2] Added: Allowing optional arguments for `scrapy.http.cookies.CookieJar.clear`
 2018-06-01 21:52:32 -03:00
Colton Herinckx 596f39600d reversed earlier change that seemed to cause Travis CI build failure 2018-06-01 21:48:43 -03:00
Colton Herinckx 12d10eec2c changed Twisted >= 17.9.0 to Twisted>=17.9.0 2018-06-01 21:48:43 -03:00
Colton Herinckx 9bd5444a42 added oxford commas to LICENSE 2018-06-01 21:48:43 -03:00
Colton Herinckx 98d9093dc7 minor grammatical fixes in CODE_OF_CONDUCT.md 2018-06-01 21:48:43 -03:00
Mikhail Korobov 3cf871c61f
Merge pull request #3281 from fbergen/gunzipperf 
[MRG+2] Improve gunzip performance for big files on Python 3
 2018-06-02 04:13:26 +05:00
Mikhail Korobov 847b50ce9c
Merge pull request #3201 from grammy-jiang/master 
[MRG+1] fix a mistake in topic spider-middleware.rst
 2018-06-02 01:09:59 +05:00
Fredrik Bergenlid 6a2d2c3b77 Improve gunzip performance for big files 2018-06-01 21:38:07 +02:00
Mikhail Korobov 13b15dc9a8
Merge pull request #3279 from lewoudar/patch-1 
Update spiders.rst
 2018-06-01 22:20:41 +05:00
Mikhail Korobov f1d87ee09a
Merge pull request #3280 from cms-/patch-1 
Minor edits to contributing.rst
 2018-06-01 20:54:39 +05:00
Chris Slothouber ecdd888ff4
Minor edits to contributing.rst 
Corrected minor grammatical issues and increased clarity of instructions.
 2018-06-01 09:25:34 -04:00
Kevin Tewouda ffa7bede17
Update spiders.rst 
I changed URLs to :class:`~scrapy.http.Request` in start_urls explanation of the default spider
 2018-05-30 06:33:18 +02:00
Vostretsov Nikita b364d27247 [MRG+1] Automatic port selection for servicies in unit tests (#3210) 
* ability to pass port as a parameter

* try to find free ports

* use environment variables to pass mock server address

* get mock server address from environment variables

* ability to select ports for proxy in runtime

* use common method for URLs from mock server

* https support

* get mock server address

* get mock address

* replace hand-written mechanism by kernel-based one

* use ephemeral ports in mockserver

* strip EOL from addresses

* use ephemeral port in proxy

* no need to restore environment as it is restored in tearDown

* decode bytes

* use mockserver address as a variable

* ability to pass address as variable

* per test-case mockserver

* use base class

* remove obsolete environment manipulation

* return usage of proxy for http cases

* common method for broking proxy auth credentials

* python version-independent url methods
 2018-05-23 21:25:50 +03:00
Konstantin Lopuhin bac1e2d47d
Merge pull request #3263 from whalebot-helmsman/no_twisted_18_4_0 
[MRG+1] Blacklist twisted version with regression
 2018-05-17 13:21:42 +03:00
Vostretsov Nikita c5ddfddb7e blacklist twisted version with regression in constraints file 2018-05-17 08:53:42 +00:00
Vostretsov Nikita 0d015e5c0f blacklist twisted version with regression 2018-05-16 09:36:07 +00:00
Konstantin Lopuhin c4f096d3a5
Merge pull request #3224 from lucywang000/better-processors-doc 
[MRG+1] Improve document about functions as processors
 2018-04-26 09:36:39 +03:00
Konstantin Lopuhin f36e1b53d4
Merge pull request #3199 from rhoboro/gcs_acl 
[MRG+1] FilesPipeline supports ACL for Google Cloud Storage
 2018-04-25 19:42:19 +03:00
Pengyu Chen e75f721c04
Added: Allowing optional arguments for `scrapy.http.cookies.CookieJar.clear` 2018-04-23 22:08:28 +08:00
Lucy Wang 57b0e6b695 improve document about functions as processors 2018-04-19 13:35:46 +08:00
rhoboro 6ef6585b5a update docs 2018-04-13 19:06:29 +09:00
rhoboro 560ee623fd set defalut value "" to FILES_STORE_GCS_ACL 2018-04-13 19:00:27 +09:00
rhoboro 464973489e Using bucket's default object ACL 2018-04-13 12:06:39 +09:00
Konstantin Lopuhin da1256a9c8
Merge pull request #3189 from stav/master 
[MRG+1] Doc: update wording for COOKIES_ENABLED
 2018-04-05 11:42:49 +03:00
grammy-jiang cb76b88331 fix a mistake in topic spider-middleware.rst 2018-04-04 05:56:05 -04:00
rhoboro 74a9c65290 update docs for support gcs acl 2018-04-03 18:20:37 +09:00
rhoboro 5254ac393b added test for gcs policy 2018-04-03 18:06:34 +09:00
rhoboro 8e8994c6b5 add acl support for gcs 2018-04-02 15:36:47 +09:00
Steven Almeroth c6d20bdd82 Doc: update wording for COOKIES_ENABLED 2018-03-27 16:21:07 -04:00
Daniel Graña 6c3970e672
Merge pull request #3153 from virmht/new_bug 
[MRG+1] Fixed bug FormRequest.from_response() clickdata ignores input[type=image]
 2018-03-21 16:32:12 -03:00
Viral Mehta dd064413a4 corrected syntax error in XPath 2018-03-19 19:28:41 +05:30
Viral Mehta a5acc9373f Resolving Comments 2018-03-19 18:19:39 +05:30
Viral Mehta e25e2afe17 Removed unnecessary print statements 2018-03-17 18:20:14 +05:30
Viral Mehta ff5f717f7a Fixed formatting issues 2018-03-17 18:17:48 +05:30
Daniel Graña 6cc6bbb5fc
Merge pull request #3166 from lucywang000/catch-tls-certificate-error 
catch CertificateError in tls verification
 2018-03-14 11:47:58 -03:00
Lucy Wang 2c58da19a6 update docstring of ScrapyClientTLSOptions 2018-03-14 09:27:59 +08:00
Lucy Wang 1a2f0193a3 fix tests on jessie 2018-03-13 19:14:52 +08:00
siulkilulki 6a7cdf9a6c [MRG+1] Add 'flv' to ignored video extensions. (#3165) 2018-03-13 10:35:27 +03:00
Lucy Wang d9e6c73fb3 revert wrong changes 2018-03-13 13:05:37 +08:00
Lucy Wang e487100987 add a test case 2018-03-13 08:59:03 +08:00
Lucy Wang 13a74d77e2 catch CertificateError in tls verification 2018-03-12 22:25:19 +08:00