RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,790 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

1187 Commits

Author SHA1 Message Date
Eugenio Lacuesta 451e7a616e
Scan callbacks/errbacks for return statements with values different than None 2019-12-16 11:30:40 -03:00
Adrián Chaves a4ef9750f9 Fix Flake8-reported issues 2019-12-13 14:32:06 +01:00
Adrián Chaves b5c4c2cae8
Keep 2 spaces between code and inline comments (#4195) 2019-12-13 14:20:48 +01:00
Andrey Rakhmatullin bfb78b8dea Add CrawlerProcess tests for ASYNCIO_ENABLED. 2019-12-13 18:12:07 +05:00
Andrey Rakhmatullin 3560123090 Rename ASYNCIO_SUPPORT to ASYNCIO_ENABLED. 2019-12-13 18:07:47 +05:00
Andrey Rakhmatullin 352ddbb336 Merge remote-tracking branch 'origin/master' into asyncio-base 2019-12-13 18:07:23 +05:00
Andrey Rahmatullin 02cdc53fb8
Add a test for a CrawlerProcess script. (#4218) 
* Add a test for a CrawlerProcess script.

* Add tests/CrawlerProcess to collect_ignore.

* Remove an extra line.

* Fix/improve conftest.py.
 2019-12-13 18:04:05 +05:00
Eugenio Lacuesta 21a2f8c6f8
Merge branch 'master' into response_follow_all 2019-12-10 17:30:59 -03:00
Adrián Chaves 1fc2b140c1
Merge branch 'master' into documentation-coverage 2019-12-05 14:43:36 +01:00
Andrey Rahmatullin 076f0764b7
Merge pull request #4121 from scrapy/remove-six-code 
Remove six-related code and __future__ imports
 2019-12-05 18:20:14 +05:00
Mikhail Korobov 250da28952
Merge pull request #4170 from mabelvj/4133-handle-start_url 
Raise error when start_url found instead of start_urls.
 2019-12-05 17:47:03 +05:00
Andrey Rakhmatullin ed34ce14c0 Add the ASYNCIO_SUPPORT setting, reshuffle other logic accordingly. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin cc19ab5439 Add tests that check asyncio support. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin b04b541372 Install the asyncio reactor only in scrapy.cmdline. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin 8d8fbddbde Switch to the released version of pytest-twisted. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin 9777639533 Run tests using the asyncio reactor. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin 1b35260625 Add a test for downloader middlewares using Deferreds. 2019-12-04 21:32:56 +05:00
Eugenio Lacuesta 62778cf23f
Request: remove restriction about errback without callback 2019-12-04 08:53:38 -03:00
Eugenio Lacuesta 2a9f5a0aef
Skip invalid links when passing SelectorLists to Response.follow_all 2019-12-03 16:01:42 -03:00
Eugenio Lacuesta e43f37fff3
Pass args/kwargs in S3DownloadHandler.from_crawler, update tests 2019-12-02 17:36:31 -03:00
Eugenio Lacuesta 3d77f74e40
Download handlers: from_crawler factory method, take crawler instead of settings in __init__ 2019-12-02 17:36:31 -03:00
Andrey Rakhmatullin 63546cbf3e Deprecate the HTTPS proxy noconnect mode. 2019-11-27 22:42:52 +05:00
Adrián Chaves 6d9ed6146d
Merge branch 'master' into remove-six-code 2019-11-25 10:34:21 +01:00
Eugenio Lacuesta 40b5cfc0a4
Item loaders: allow single-argument processors (unbound methods) 2019-11-22 20:47:22 -03:00
Eugenio Lacuesta 6fa8f06b64
Merge remote-tracking branch 'upstream/master' into response_follow_all 2019-11-22 13:05:06 -03:00
Eugenio Lacuesta 5bab3c0261
Merge remote-tracking branch 'upstream/master' into remove-six-code 2019-11-22 12:12:29 -03:00
Mikhail Korobov 16e0636dcf
Merge pull request #4186 from Gallaecio/lgtm 
Remove unused imports
 2019-11-22 12:28:29 +05:00
Mabel Villalba 070b3a4e84
Merge branch 'master' into 4133-handle-start_url 2019-11-21 17:10:31 +01:00
Mabel Villalba 1718e450ef [start_url] Fixes #4133: Raise AttributeError error when empty 'start_urls' and 'start_url' found. Added test. 2019-11-21 17:06:47 +01:00
Mikhail Korobov d2b73b8e21
Merge pull request #4114 from scrapy/remove-py2-tests 
Remove py2 tests
 2019-11-21 18:22:10 +05:00
Adrián Chaves a2bf340bab Remove unused imports 2019-11-21 14:21:17 +01:00
Eugenio Lacuesta 714e0d3ac9
Merge remote-tracking branch 'upstream/master' into remove-six-code 2019-11-20 12:41:38 -03:00
Mikhail Korobov 65e2fb7d89
Merge pull request #4153 from Gallaecio/lgtm 
MutableChain: return self from __iter__
 2019-11-20 19:46:20 +05:00
Mikhail Korobov e96ca89a1d
Merge pull request #4174 from Gallaecio/flake8 
Add missing blank lines between functions and classes
 2019-11-20 19:34:04 +05:00
Eugenio Lacuesta 6781d2f5b2
Update sample file references 2019-11-20 09:58:25 -03:00
Eugenio Lacuesta 48223c107a
Merge remote-tracking branch 'upstream/master' into response_follow_all 2019-11-20 09:41:42 -03:00
Eugenio Lacuesta b602c61e1c
[Test] Rename outdated sample files 2019-11-20 09:38:54 -03:00
Eugenio Lacuesta 6f6426559d
Merge remote-tracking branch 'upstream/remove-py2-tests' into remove-six-code 2019-11-19 11:04:46 -03:00
Eugenio Lacuesta 05785c1c17
Merge remote-tracking branch 'upstream/master' into remove-six-code 2019-11-19 11:01:34 -03:00
Andrey Rahmatullin 78ad01632f Fix flake8 problems in PR #3989 (#4176) 2019-11-19 10:43:30 +01:00
Andrey Rakhmatullin beb7d80d6a Add a comment about the noconnect tests. 2019-11-19 13:52:49 +05:00
Andrey Rakhmatullin 922a66cf07 Fix or ignore flake8 problems. 2019-11-19 13:52:49 +05:00
Andrey Rakhmatullin a7b640991d Rename tests/py3-ignores.txt to tests/ignores.txt. 2019-11-19 13:52:49 +05:00
Andrey Rakhmatullin 8b730a3670 Use self.proc.communicate() after killing mitmdump. 2019-11-19 13:52:49 +05:00
Andrey Rakhmatullin 5970d00eb9 Only xfail test_https_connect_tunnel_error on 3.6+. 2019-11-19 13:52:49 +05:00
Andrey Rakhmatullin 5080180c75 Improve the test_https_tunnel_without_leak_proxy_authorization_header change. 2019-11-19 13:52:49 +05:00
Andrey Rakhmatullin c4ef950efd Use an older mitmproxy for py3.5. 2019-11-19 13:52:49 +05:00
Andrey Rakhmatullin 3ec6960732 Fix test_proxy_connect.py for py3.5. 2019-11-19 13:52:49 +05:00
Andrey Rakhmatullin c327ad9ba6 Remove an unused six import. 2019-11-19 13:52:49 +05:00
Andrey Rakhmatullin cbb6d0c6a7 Mark failing proxy tests. 2019-11-19 13:52:49 +05:00
Andrey Rakhmatullin f066257e95 Restore tests/test_proxy_connect.py and update it to modern mitmproxy. 2019-11-19 13:52:49 +05:00
Andrey Rakhmatullin e18014d84d Remove Python 2-only tests. 2019-11-19 13:52:49 +05:00
Andrey Rahmatullin 25cd7ac7ab
Merge pull request #4094 from victor-torres/invalid-url 
Improve URL schema validation on scrapy.Request initialization
 2019-11-19 13:50:08 +05:00
Andrey Rahmatullin 3408b757c1
Merge pull request #3989 from elacuesta/logformatter_error_method 
LogFormatter improvements
 2019-11-19 13:44:43 +05:00
Adrián Chaves f261cf65e9 Add missing blank lines between functions and classes 
Also fixed 2 unrelated Flake8 issues
 2019-11-18 17:16:09 +01:00
Adrián Chaves 99d8b05a0b Deprecate scrapy.utils.python.MutableChain.next 2019-11-18 10:58:47 +01:00
Marc Hernandez Cabot 0e252f5a13 fix E711 and E713 2019-11-15 19:12:43 +01:00
Mikhail Korobov 550d8046fd
Merge pull request #4157 from Gallaecio/flake8 
Add missing whitespace after ‘,’, ‘;’ or ‘:’
 2019-11-14 23:42:06 +05:00
Andrey Rakhmatullin 3b2289ad01 Rename test_non_str_url_py2 to test_bytes_url. 2019-11-14 22:53:28 +05:00
Andrey Rakhmatullin fe3a121f13 Use kwargs when calling get_func_args. 2019-11-14 22:50:53 +05:00
Adrián Chaves 058bdda0af Improve the performance of the DOWNLOAD_DELAY test 2019-11-14 16:53:20 +01:00
Andrey Rakhmatullin e291460db6 Fix flake8-detected errors. 2019-11-14 15:24:37 +05:00
Adrián Chaves 1a4a77d49f Remove Python 2 check from MutableChainTest 2019-11-14 10:24:31 +01:00
Andrey Rakhmatullin 87c23ba22d Remove Py2-only code that checks sys.version_info. 2019-11-14 13:45:39 +05:00
Andrey Rakhmatullin a138fb05d4 Replace to_native_str calls with to_unicode. 2019-11-14 13:45:39 +05:00
Andrey Rakhmatullin 92ffd2f249 Simplify some more imports. 2019-11-14 13:45:39 +05:00
Andrey Rakhmatullin 85e79ae792 Remove cStringIO imports. 2019-11-14 13:45:39 +05:00
Andrey Rakhmatullin 1d7c8cb0b1 Remove six.PY2 and six.PY3 conditionals. 2019-11-14 13:45:37 +05:00
Adrián Chaves 33ef24c757 Add missing whitespace after ‘,’, ‘;’ or ‘:’ 2019-11-13 10:52:05 +01:00
Andrey Rahmatullin c911e80209
Merge pull request #4088 from ammarnajjar/4086-constructor-initializer-docs 
docs: use __init__ method instead of constructor
 2019-11-12 21:48:16 +05:00
Andrey Rahmatullin c91390558d
Merge pull request #4074 from purvaudai/master 
Added Pathlib.Path support: Issue #3731
 2019-11-12 21:43:28 +05:00
purvaudai 4b8b0345e5 Mades Changes as per review 2019-11-12 18:17:15 +05:30
Andrey Rahmatullin 93385e647a
Merge pull request #4123 from elacuesta/utils-local-cache-limit 
Fix LocalCache limit issue, add tests
 2019-11-12 16:05:22 +05:00
purvaudai 50eaabe1fc Added Test 2019-11-11 20:00:26 +05:30
purvaudai f39ff49458 Added Test 2019-11-11 18:54:21 +05:30
purvaudai 970c3be160 Added Test 2019-11-11 18:34:15 +05:30
purvaudai 9e6e2dde2b Adding test 2019-11-11 16:10:37 +05:30
purvaudai 0042c389eb Adding test 2019-11-11 15:57:58 +05:30
purvaudai 084a1cda6d Adding test 2019-11-11 15:41:00 +05:30
Eugenio Lacuesta 07fa768d33
Merge branch 'remove-six-code' of github.com:scrapy/scrapy into remove-six-code 2019-11-08 22:17:12 -03:00
Eugenio Lacuesta b6bbb28197
PEP8 adjustments 2019-11-08 22:13:03 -03:00
Eugenio Lacuesta 342e3b5bd5
Merge remote-tracking branch 'upstream/master' into remove-six-code 2019-11-08 12:29:15 -03:00
Adrián Chaves 1df5755699 Set the bases for testing code examples from the documentation 2019-11-08 16:00:10 +01:00
elacuesta 44f19df311
[test] Update mitmproxy version 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2019-11-08 11:32:50 -03:00
Marc Hernández c377c14e32 Fix W391 Blank line at end of file (#4137) 2019-11-07 17:47:35 +01:00
elacuesta fe31695ba0
Remove unused import (urllib.parse.unquote) 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2019-11-05 15:36:19 -03:00
Eugenio Lacuesta 613c66a034
Do not override built-in max function 2019-11-05 09:45:51 -03:00
elacuesta fed9fbe62d
Update tests/test_utils_datatypes.py 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2019-11-04 15:34:27 -03:00
Eugenio Lacuesta 439a3e59b8
Fix scrapy.utils.datatypes.LocalCache limit issue 2019-11-04 10:35:58 -03:00
Eugenio Lacuesta 00b793dc59
Remove elluding six occurrences 2019-11-03 12:26:38 -03:00
Eugenio Lacuesta 5797aefd4c
Remove six.assertCountEqual 2019-11-03 12:18:35 -03:00
Eugenio Lacuesta 586b25d27e
Remove six types 2019-11-03 12:10:37 -03:00
Eugenio Lacuesta 7f3cb05d8e
Remove metaclass-related six code 2019-11-03 12:03:02 -03:00
Eugenio Lacuesta cca1959c07
Merge remote-tracking branch 'upstream/remove-py2-tests' into remove-six-code 2019-11-03 01:20:08 -03:00
Eugenio Lacuesta d72444b9c8
Remove more six imports 2019-11-03 01:11:28 -03:00
Eugenio Lacuesta eaeaa40b99
Remove six.PY* checks 2019-11-03 01:08:08 -03:00
Eugenio Lacuesta 5d8abdde59
Remove six.text_type from tests 2019-11-03 01:01:23 -03:00
Eugenio Lacuesta 54a786b102
Remove six imports 2019-11-03 00:58:47 -03:00
Eugenio Lacuesta 1aba513693
Remove six.iter* occurrences 2019-11-03 00:26:44 -03:00
Eugenio Lacuesta 5ab0f189ce
Remove six.moves occurrences from tests 2019-11-03 00:01:09 -03:00