RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
8,050 Commits 42 Branches 132 Tags 168 MiB
Commit Graph

1302 Commits

Author SHA1 Message Date
Andrey Rakhmatullin 7ccb169a27 Split a long test in test_engine.py into three. 2019-12-18 19:41:16 +05:00
Mikhail Korobov e90f276903
Merge pull request #4198 from wRAR/deprecate-noconnect 
Deprecate the HTTPS proxy noconnect mode.
 2019-12-18 16:11:24 +05:00
Adrián Chaves 1f689b50c6 Merge remote-tracking branch 'origin/documentation-coverage' into documentation-coverage 2019-12-18 12:09:34 +01:00
Adrián Chaves 174769a3f0 Use a better name for the LxmlLinkExtractor subclassing test 2019-12-18 12:09:03 +01:00
Andrey Rakhmatullin ac302c3f61 Fix a flake8 problem. 2019-12-18 15:43:05 +05:00
Andrey Rakhmatullin bb2ff13e4c Skip Http10ProxyTestCase.test_download_with_proxy_https_noconnect 2019-12-18 15:39:08 +05:00
Andrey Rakhmatullin 2d92a39003 Restore test_download_with_proxy_https_noconnect, check for a warning there. 2019-12-18 12:07:08 +05:00
Adrián Chaves 607815dfc2
Merge branch 'master' into documentation-coverage 2019-12-17 16:54:27 +01:00
Andrey Rahmatullin bb3f164280
Merge pull request #4236 from wRAR/pipeline-tests 
Add simple tests for pipelines.
 2019-12-17 19:38:06 +05:00
Marc Hernández 63cf5c75c8 Fix E502: backslash is redundant between brackets (#4238) 2019-12-17 13:53:15 +01:00
Andrey Rakhmatullin 5980d3bbff Add simple tests for pipelines. 2019-12-16 22:41:28 +05:00
Eugenio Lacuesta 451e7a616e
Scan callbacks/errbacks for return statements with values different than None 2019-12-16 11:30:40 -03:00
Adrián Chaves a4ef9750f9 Fix Flake8-reported issues 2019-12-13 14:32:06 +01:00
Adrián Chaves b5c4c2cae8
Keep 2 spaces between code and inline comments (#4195) 2019-12-13 14:20:48 +01:00
Andrey Rakhmatullin bfb78b8dea Add CrawlerProcess tests for ASYNCIO_ENABLED. 2019-12-13 18:12:07 +05:00
Andrey Rakhmatullin 3560123090 Rename ASYNCIO_SUPPORT to ASYNCIO_ENABLED. 2019-12-13 18:07:47 +05:00
Andrey Rakhmatullin 352ddbb336 Merge remote-tracking branch 'origin/master' into asyncio-base 2019-12-13 18:07:23 +05:00
Andrey Rahmatullin 02cdc53fb8
Add a test for a CrawlerProcess script. (#4218) 
* Add a test for a CrawlerProcess script.

* Add tests/CrawlerProcess to collect_ignore.

* Remove an extra line.

* Fix/improve conftest.py.
 2019-12-13 18:04:05 +05:00
Eugenio Lacuesta 21a2f8c6f8
Merge branch 'master' into response_follow_all 2019-12-10 17:30:59 -03:00
Adrián Chaves 1fc2b140c1
Merge branch 'master' into documentation-coverage 2019-12-05 14:43:36 +01:00
Andrey Rahmatullin 076f0764b7
Merge pull request #4121 from scrapy/remove-six-code 
Remove six-related code and __future__ imports
 2019-12-05 18:20:14 +05:00
Mikhail Korobov 250da28952
Merge pull request #4170 from mabelvj/4133-handle-start_url 
Raise error when start_url found instead of start_urls.
 2019-12-05 17:47:03 +05:00
Andrey Rakhmatullin ed34ce14c0 Add the ASYNCIO_SUPPORT setting, reshuffle other logic accordingly. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin cc19ab5439 Add tests that check asyncio support. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin b04b541372 Install the asyncio reactor only in scrapy.cmdline. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin 8d8fbddbde Switch to the released version of pytest-twisted. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin 9777639533 Run tests using the asyncio reactor. 2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin 1b35260625 Add a test for downloader middlewares using Deferreds. 2019-12-04 21:32:56 +05:00
Eugenio Lacuesta 62778cf23f
Request: remove restriction about errback without callback 2019-12-04 08:53:38 -03:00
Eugenio Lacuesta 2a9f5a0aef
Skip invalid links when passing SelectorLists to Response.follow_all 2019-12-03 16:01:42 -03:00
Eugenio Lacuesta e43f37fff3
Pass args/kwargs in S3DownloadHandler.from_crawler, update tests 2019-12-02 17:36:31 -03:00
Eugenio Lacuesta 3d77f74e40
Download handlers: from_crawler factory method, take crawler instead of settings in __init__ 2019-12-02 17:36:31 -03:00
Andrey Rakhmatullin 63546cbf3e Deprecate the HTTPS proxy noconnect mode. 2019-11-27 22:42:52 +05:00
Adrián Chaves 6d9ed6146d
Merge branch 'master' into remove-six-code 2019-11-25 10:34:21 +01:00
Eugenio Lacuesta 03af8885ff
LogFormatter.download_error 2019-11-23 20:02:44 -03:00
Eugenio Lacuesta 4756e7c587
LogFormatter.spider_error 2019-11-23 19:33:29 -03:00
Eugenio Lacuesta facb926542
Remove quotes from item_error message 2019-11-23 19:16:41 -03:00
Eugenio Lacuesta 7a7d13b112
Rename LogFormatter.error to item_error 2019-11-23 19:12:03 -03:00
Eugenio Lacuesta 40b5cfc0a4
Item loaders: allow single-argument processors (unbound methods) 2019-11-22 20:47:22 -03:00
Eugenio Lacuesta 6fa8f06b64
Merge remote-tracking branch 'upstream/master' into response_follow_all 2019-11-22 13:05:06 -03:00
Eugenio Lacuesta 5bab3c0261
Merge remote-tracking branch 'upstream/master' into remove-six-code 2019-11-22 12:12:29 -03:00
Mikhail Korobov 16e0636dcf
Merge pull request #4186 from Gallaecio/lgtm 
Remove unused imports
 2019-11-22 12:28:29 +05:00
Mabel Villalba 070b3a4e84
Merge branch 'master' into 4133-handle-start_url 2019-11-21 17:10:31 +01:00
Mabel Villalba 1718e450ef [start_url] Fixes #4133: Raise AttributeError error when empty 'start_urls' and 'start_url' found. Added test. 2019-11-21 17:06:47 +01:00
Mikhail Korobov d2b73b8e21
Merge pull request #4114 from scrapy/remove-py2-tests 
Remove py2 tests
 2019-11-21 18:22:10 +05:00
Adrián Chaves a2bf340bab Remove unused imports 2019-11-21 14:21:17 +01:00
Eugenio Lacuesta 714e0d3ac9
Merge remote-tracking branch 'upstream/master' into remove-six-code 2019-11-20 12:41:38 -03:00
Mikhail Korobov 65e2fb7d89
Merge pull request #4153 from Gallaecio/lgtm 
MutableChain: return self from __iter__
 2019-11-20 19:46:20 +05:00
Mikhail Korobov e96ca89a1d
Merge pull request #4174 from Gallaecio/flake8 
Add missing blank lines between functions and classes
 2019-11-20 19:34:04 +05:00
Eugenio Lacuesta 6781d2f5b2
Update sample file references 2019-11-20 09:58:25 -03:00
Eugenio Lacuesta 48223c107a
Merge remote-tracking branch 'upstream/master' into response_follow_all 2019-11-20 09:41:42 -03:00
Eugenio Lacuesta b602c61e1c
[Test] Rename outdated sample files 2019-11-20 09:38:54 -03:00
Eugenio Lacuesta 6f6426559d
Merge remote-tracking branch 'upstream/remove-py2-tests' into remove-six-code 2019-11-19 11:04:46 -03:00
Eugenio Lacuesta 05785c1c17
Merge remote-tracking branch 'upstream/master' into remove-six-code 2019-11-19 11:01:34 -03:00
Andrey Rahmatullin 78ad01632f Fix flake8 problems in PR #3989 (#4176) 2019-11-19 10:43:30 +01:00
Andrey Rakhmatullin beb7d80d6a Add a comment about the noconnect tests. 2019-11-19 13:52:49 +05:00
Andrey Rakhmatullin 922a66cf07 Fix or ignore flake8 problems. 2019-11-19 13:52:49 +05:00
Andrey Rakhmatullin a7b640991d Rename tests/py3-ignores.txt to tests/ignores.txt. 2019-11-19 13:52:49 +05:00
Andrey Rakhmatullin 8b730a3670 Use self.proc.communicate() after killing mitmdump. 2019-11-19 13:52:49 +05:00
Andrey Rakhmatullin 5970d00eb9 Only xfail test_https_connect_tunnel_error on 3.6+. 2019-11-19 13:52:49 +05:00
Andrey Rakhmatullin 5080180c75 Improve the test_https_tunnel_without_leak_proxy_authorization_header change. 2019-11-19 13:52:49 +05:00
Andrey Rakhmatullin c4ef950efd Use an older mitmproxy for py3.5. 2019-11-19 13:52:49 +05:00
Andrey Rakhmatullin 3ec6960732 Fix test_proxy_connect.py for py3.5. 2019-11-19 13:52:49 +05:00
Andrey Rakhmatullin c327ad9ba6 Remove an unused six import. 2019-11-19 13:52:49 +05:00
Andrey Rakhmatullin cbb6d0c6a7 Mark failing proxy tests. 2019-11-19 13:52:49 +05:00
Andrey Rakhmatullin f066257e95 Restore tests/test_proxy_connect.py and update it to modern mitmproxy. 2019-11-19 13:52:49 +05:00
Andrey Rakhmatullin e18014d84d Remove Python 2-only tests. 2019-11-19 13:52:49 +05:00
Andrey Rahmatullin 25cd7ac7ab
Merge pull request #4094 from victor-torres/invalid-url 
Improve URL schema validation on scrapy.Request initialization
 2019-11-19 13:50:08 +05:00
Andrey Rahmatullin 3408b757c1
Merge pull request #3989 from elacuesta/logformatter_error_method 
LogFormatter improvements
 2019-11-19 13:44:43 +05:00
Adrián Chaves f261cf65e9 Add missing blank lines between functions and classes 
Also fixed 2 unrelated Flake8 issues
 2019-11-18 17:16:09 +01:00
Adrián Chaves 99d8b05a0b Deprecate scrapy.utils.python.MutableChain.next 2019-11-18 10:58:47 +01:00
Marc Hernandez Cabot 0e252f5a13 fix E711 and E713 2019-11-15 19:12:43 +01:00
Mikhail Korobov 550d8046fd
Merge pull request #4157 from Gallaecio/flake8 
Add missing whitespace after ‘,’, ‘;’ or ‘:’
 2019-11-14 23:42:06 +05:00
Andrey Rakhmatullin 3b2289ad01 Rename test_non_str_url_py2 to test_bytes_url. 2019-11-14 22:53:28 +05:00
Andrey Rakhmatullin fe3a121f13 Use kwargs when calling get_func_args. 2019-11-14 22:50:53 +05:00
Adrián Chaves 058bdda0af Improve the performance of the DOWNLOAD_DELAY test 2019-11-14 16:53:20 +01:00
Andrey Rakhmatullin e291460db6 Fix flake8-detected errors. 2019-11-14 15:24:37 +05:00
Adrián Chaves 1a4a77d49f Remove Python 2 check from MutableChainTest 2019-11-14 10:24:31 +01:00
Andrey Rakhmatullin 87c23ba22d Remove Py2-only code that checks sys.version_info. 2019-11-14 13:45:39 +05:00
Andrey Rakhmatullin a138fb05d4 Replace to_native_str calls with to_unicode. 2019-11-14 13:45:39 +05:00
Andrey Rakhmatullin 92ffd2f249 Simplify some more imports. 2019-11-14 13:45:39 +05:00
Andrey Rakhmatullin 85e79ae792 Remove cStringIO imports. 2019-11-14 13:45:39 +05:00
Andrey Rakhmatullin 1d7c8cb0b1 Remove six.PY2 and six.PY3 conditionals. 2019-11-14 13:45:37 +05:00
Adrián Chaves 33ef24c757 Add missing whitespace after ‘,’, ‘;’ or ‘:’ 2019-11-13 10:52:05 +01:00
Andrey Rahmatullin c911e80209
Merge pull request #4088 from ammarnajjar/4086-constructor-initializer-docs 
docs: use __init__ method instead of constructor
 2019-11-12 21:48:16 +05:00
Andrey Rahmatullin c91390558d
Merge pull request #4074 from purvaudai/master 
Added Pathlib.Path support: Issue #3731
 2019-11-12 21:43:28 +05:00
purvaudai 4b8b0345e5 Mades Changes as per review 2019-11-12 18:17:15 +05:30
Andrey Rahmatullin 93385e647a
Merge pull request #4123 from elacuesta/utils-local-cache-limit 
Fix LocalCache limit issue, add tests
 2019-11-12 16:05:22 +05:00
purvaudai 50eaabe1fc Added Test 2019-11-11 20:00:26 +05:30
purvaudai f39ff49458 Added Test 2019-11-11 18:54:21 +05:30
purvaudai 970c3be160 Added Test 2019-11-11 18:34:15 +05:30
purvaudai 9e6e2dde2b Adding test 2019-11-11 16:10:37 +05:30
purvaudai 0042c389eb Adding test 2019-11-11 15:57:58 +05:30
purvaudai 084a1cda6d Adding test 2019-11-11 15:41:00 +05:30
Eugenio Lacuesta 07fa768d33
Merge branch 'remove-six-code' of github.com:scrapy/scrapy into remove-six-code 2019-11-08 22:17:12 -03:00
Eugenio Lacuesta b6bbb28197
PEP8 adjustments 2019-11-08 22:13:03 -03:00
Eugenio Lacuesta 342e3b5bd5
Merge remote-tracking branch 'upstream/master' into remove-six-code 2019-11-08 12:29:15 -03:00
Adrián Chaves 1df5755699 Set the bases for testing code examples from the documentation 2019-11-08 16:00:10 +01:00
elacuesta 44f19df311
[test] Update mitmproxy version 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2019-11-08 11:32:50 -03:00
Marc Hernández c377c14e32 Fix W391 Blank line at end of file (#4137) 2019-11-07 17:47:35 +01:00
elacuesta fe31695ba0
Remove unused import (urllib.parse.unquote) 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2019-11-05 15:36:19 -03:00
Eugenio Lacuesta 613c66a034
Do not override built-in max function 2019-11-05 09:45:51 -03:00
elacuesta fed9fbe62d
Update tests/test_utils_datatypes.py 
Co-Authored-By: Adrián Chaves <adrian@chaves.io>
 2019-11-04 15:34:27 -03:00
Eugenio Lacuesta 439a3e59b8
Fix scrapy.utils.datatypes.LocalCache limit issue 2019-11-04 10:35:58 -03:00
Eugenio Lacuesta 00b793dc59
Remove elluding six occurrences 2019-11-03 12:26:38 -03:00
Eugenio Lacuesta 5797aefd4c
Remove six.assertCountEqual 2019-11-03 12:18:35 -03:00
Eugenio Lacuesta 586b25d27e
Remove six types 2019-11-03 12:10:37 -03:00
Eugenio Lacuesta 7f3cb05d8e
Remove metaclass-related six code 2019-11-03 12:03:02 -03:00
Eugenio Lacuesta cca1959c07
Merge remote-tracking branch 'upstream/remove-py2-tests' into remove-six-code 2019-11-03 01:20:08 -03:00
Eugenio Lacuesta d72444b9c8
Remove more six imports 2019-11-03 01:11:28 -03:00
Eugenio Lacuesta eaeaa40b99
Remove six.PY* checks 2019-11-03 01:08:08 -03:00
Eugenio Lacuesta 5d8abdde59
Remove six.text_type from tests 2019-11-03 01:01:23 -03:00
Eugenio Lacuesta 54a786b102
Remove six imports 2019-11-03 00:58:47 -03:00
Eugenio Lacuesta 1aba513693
Remove six.iter* occurrences 2019-11-03 00:26:44 -03:00
Eugenio Lacuesta 5ab0f189ce
Remove six.moves occurrences from tests 2019-11-03 00:01:09 -03:00
Eugenio Lacuesta c0bfaef37a
Remove __future__ imports from tests 2019-11-02 23:27:04 -03:00
Andrey Rakhmatullin 350aa67c3d Rename tests/py3-ignores.txt to tests/ignores.txt. 2019-11-01 19:52:57 +05:00
Andrey Rakhmatullin 4b0cdf7f3e Use self.proc.communicate() after killing mitmdump. 2019-11-01 19:51:52 +05:00
Andrey Rakhmatullin 3c9963ab04 Only xfail test_https_connect_tunnel_error on 3.6+. 2019-11-01 19:47:12 +05:00
Andrey Rakhmatullin e0c5c72496 Improve the test_https_tunnel_without_leak_proxy_authorization_header change. 2019-11-01 19:46:19 +05:00
Andrey Rakhmatullin 5eb01b617d Use an older mitmproxy for py3.5. 2019-10-31 23:21:30 +05:00
Andrey Rakhmatullin a5eb59b92d Fix test_proxy_connect.py for py3.5. 2019-10-31 23:21:14 +05:00
Adrián Chaves f02c3d1dcf
Use communicate() instead of wait() after killing the mock server (#4095) 2019-10-31 13:31:33 +01:00
Andrey Rakhmatullin 7299e91b1f Remove Py2-only code that checks sys.version_info. 2019-10-31 15:22:53 +05:00
Andrey Rakhmatullin 397e883556 Replace to_native_str calls with to_unicode. 2019-10-31 15:22:53 +05:00
Andrey Rakhmatullin 75b1d051d9 Simplify some more imports. 2019-10-31 15:22:45 +05:00
Andrey Rakhmatullin 3ac4b430ae Remove an unused six import. 2019-10-31 15:20:28 +05:00
Andrey Rakhmatullin cea2f5e244 Remove cStringIO imports. 2019-10-31 15:07:54 +05:00
Andrey Rakhmatullin bbd9f4be90 Remove six.PY2 and six.PY3 conditionals. 2019-10-31 15:07:10 +05:00
Andrey Rakhmatullin 186f9d88ac Fix the skip message for test_download_gzip_response. 2019-10-31 15:03:44 +05:00
Andrey Rakhmatullin 439e37fc7b Mark failing proxy tests. 2019-10-31 15:03:44 +05:00
Andrey Rakhmatullin b0d6f4917d Restore tests/test_proxy_connect.py and update it to modern mitmproxy. 2019-10-31 15:03:44 +05:00
Andrey Rakhmatullin b44bd6f825 Remove Python 2-only tests. 2019-10-31 15:03:09 +05:00
Adrián Chaves 15c55d0c1d
Remove LevelDB support (#4112) 2019-10-31 10:47:29 +01:00
Andrey Rahmatullin 229e722a03 Initial Python 2 removal (#4091) 2019-10-31 10:46:02 +01:00
Benjamin Ooghe-Tabanou 6d6da78eda Add a keep_fragments parameter to the request_fingerprint function (#4104) 2019-10-30 09:13:36 +01:00
Amardeep Bhowmick 66cbceeb0a Fix redirection error when the Location header value starts with 3 slashes (#4042) 2019-10-30 09:09:12 +01:00
Andrey Rahmatullin 18b808b2e9
Merge pull request #4092 from further-reading/master 
Add Python 3.8 official support
 2019-10-29 16:30:58 +05:00
Roy 93e3dc1b82 [test_downloadermiddleware_httpcache.py] Cleaning text 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-28 16:12:03 +00:00
Roy b73d217de5 [test_downloadermiddleware_httpcache.py] Fixing pytest mark behaviour 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-28 12:55:54 +00:00
Roy c51fb959e2 [test_downloadermiddleware_httpcache] Fixing pytest skip behaviour 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-28 12:36:54 +00:00
Roy 4432136fff [test_downloadermiddleware_httpcache] Fixing pytest skip behaviour 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-28 12:22:21 +00:00
Roy 16bb3ac20d [test_downloadermiddleware_httpcache] Using skipif approach 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-28 11:24:09 +00:00
elacuesta 7731814cc2 ItemLoader: improve handling of initial item (#4036) 2019-10-28 10:53:53 +01:00
Roy 20ea912513 [test_downloadermiddleware_httpcache] Making xfails more informative 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-27 18:52:01 +00:00
Roy 70b2854590 [test_downloadermiddleware_httpcache] Making xfails more informative 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-27 18:51:26 +00:00
Roy b3df0a8415 [test_downloadermiddleware_httpcache] Adding xfails to impacted tests following hack fix 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-27 18:28:47 +00:00
Roy 11942c436c [test_downloadermiddleware_httpcache] Trying hack to handle systemerror whjen importing leveldb 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-27 18:02:13 +00:00
Roy deacd34c8d [test_downloadermiddleware_httpcache] Attempting to add xfail for leveldb related tests 
https://github.com/scrapy/scrapy/issues/4085
 2019-10-27 17:39:47 +00:00
Roy Healy 4068797558
Update test_downloadermiddleware_httpcache.py 
Adding xfail denoting that leveldb is not supported in 3.8
 2019-10-27 17:02:17 +00:00
Victor Torres f701f5b0db fix #2552 by improving request schema check on its initialization 2019-10-22 10:48:02 -03:00
Ammar Najjar 7a84a4bdba
docs: use "constructor" for from_settings() & rom_crawler() factory methods 
Issue #4086
 2019-10-22 15:31:34 +02:00
Adrián Chaves 1d5c270ce8
Fix dangling file descriptor in FeedExporter when FEED_STORE_EMPTY is False (#4023) 2019-10-22 15:12:52 +02:00
purvaudai 4184bac068 Added Pathlib.Path test 2019-10-22 16:57:14 +05:30
purvaudai 85f56a92f0 Added Pathlib.Path test 2019-10-22 16:43:17 +05:30
purvaudai 7031e3a124 Added Pathlib.Path test 2019-10-22 16:31:14 +05:30
purvaudai cd4c211f4b Added Pathlib.Path test 2019-10-22 15:38:06 +05:30
purvaudai a776554282 Added Pathlib.Path test 2019-10-22 15:31:55 +05:30
purvaudai ad96d6ef59 Added Pathlib.Path test correctly 2019-10-22 14:53:59 +05:30
purvaudai 2ee38e8ddb Added Pathlib.Path test 2019-10-22 14:43:47 +05:30
Ammar Najjar da8cd9448d
docs: always surround __init__ with `` in docs 
Issue #4086
 2019-10-21 19:56:58 +02:00
Ammar Najjar 68a7d05ed8
docs: use __init__ method instead of constructor 
Issue #4086
 2019-10-21 15:42:24 +02:00
Eugenio Lacuesta e1fa1fd8ad
TextResponse.follow_all: skip invalid links 2019-10-10 00:37:30 -03:00
Eugenio Lacuesta 5f168cd459
Response.follow_all 2019-10-08 16:32:56 -03:00
Eugenio Lacuesta e0fabab5cc
Fix TypeError when using DummyStatsCollector 2019-10-01 18:03:22 -03:00
Eugenio Lacuesta 07a31b13db
Update LogFormatter tests 2019-10-01 17:55:57 -03:00
Adrián Chaves 7f4f98fd38 Provide complete API documentation coverage of scrapy.linkextractors 2019-09-30 18:22:28 +02:00
s-sanjay 2c14692e60 remove .keys() to avoid creating a tmp list/keyview obj (#4031) 
Also add --verbose and --nolinks for code coverage
 2019-09-27 09:56:43 +02:00
Mikhail Korobov 31c631f9ee
Merge pull request #3999 from Gallaecio/documentation-coverage 
Provide complete API documentation coverage of scrapy.item
 2019-09-25 16:13:13 +01:00
Adrián Chaves 1236e9e81e Provide complete API documentation coverage of scrapy.item 2019-09-25 13:24:56 +02:00
Mikhail Korobov 1364e92007
Merge pull request #4015 from elacuesta/remove_deprecated_xlib 
Remove deprecated xlib module
 2019-09-20 15:28:07 +01:00
Mikhail Korobov 17c0d72d04
Merge pull request #4016 from elacuesta/rule_default_link_extractor 
Crawling rules: make link extractors optional
 2019-09-20 15:27:17 +01:00
Eugenio Lacuesta 27436cbbc9
[test] LogFormatter.error 2019-09-16 12:12:10 -03:00
watsta 0b52fa6ca9 LogFormatter: Add the ability to skip log messages (#3987) 2019-09-16 14:12:04 +02:00
OmarFarrag 13735bcf34 Disallow media extensions unregistered with IANA (#3954) 
Co-Authored-By: s-sanjay <sanjay537@gmail.com>
 2019-09-16 14:04:06 +02:00
Eugenio Lacuesta 21ad8e20b9
Crawling rules: make link extractors optional 2019-09-13 17:32:42 -03:00
Eugenio Lacuesta c5612f387b
Remove deprecated xlib module 2019-09-13 14:32:05 -03:00
Mikhail Korobov 534de7395d
Merge pull request #3988 from elacuesta/contracts_cb_kwargs 
CallbackKeywordArgumentsContract
 2019-09-08 04:23:16 +05:00
OmarFarrag 0e8770a2f4 test for files pipeline ftp store 2019-09-06 15:47:57 +02:00
Eugenio Lacuesta b92b114633
[test] cb_kwargs contract 2019-08-31 02:44:09 -03:00
Marc Hernández ace2df3d14 Fix JSONRequest naming (#3982) 2019-08-30 11:03:44 +02:00
Mikhail Korobov ede91478e5
Merge pull request #3966 from anubhavp28/robotstxt_useragent 
Adds ROBOTSTXT_USER_AGENT setting
 2019-08-28 22:00:17 +05:00
Anubhav Patel 3a7b949d6d Adds integration with Protego robots.txt parser (#3935) 2019-08-27 09:41:31 +02:00
Anubhav Patel 00fe05e536 adds ROBOTSTXT_USER_AGENT setting 2019-08-19 09:24:16 +05:30
Andrey Rakhmatullin aaa5229e5d Fixes and improvements for DOWNLOADER_CLIENT_TLS_CIPHERS. 2019-08-13 16:56:26 +05:00
Andrey Rakhmatullin 9a8edf2bf1 Tests for setting SSL ciphers. 2019-08-13 16:53:19 +05:00
Mikhail Korobov 73d1b4b748
Merge pull request #3939 from Gallaecio/improve-scrapes-contract 
Report all missing fields when a scrapes contract fails
 2019-08-08 13:49:09 +05:00
Mikhail Korobov b4556d6508
Merge pull request #3941 from starrify/ftp-storage-uri-unquote 
[MRG+1] Added: Properly handling quoted passwords in FEED_URI for FTP
 2019-08-08 13:38:17 +05:00
Adrián Chaves 9119798a5c Add test coverage for contract failures involving multiple missing fields 2019-08-08 09:52:03 +02:00
Marc Hernández d76b6944c9 Create Request from curl command (#3862) 2019-08-08 09:43:42 +02:00
elacuesta 5dbeece8da [MRG+1] Drop py34 support - Update CI envs (#3892) 
* Drop py34 support

* Travis experiments

* More Travis experiments

* Bump Twisted version for py35+ (stretch)

* Remove Debian build

* Remove pinned lxml for Py34

* Fix merge error

* Remove unused tox env

* Add environment with pinned versions for py36

* Bump minimum Twisted version in py27; Envs with pinned versions for py27 and py35

* Add botocore as extra dep for py27 tests

* Update requirements-py2.txt

* Add botocore and Pillow as extra dependencies
 2019-08-07 12:36:52 +05:00
Pengyu Chen 7b755a41a1
Added: Properly handling quoted passwords in FEED_URI for FTP 2019-08-06 15:19:57 +01:00
Anubhav Patel 8e813953bd [MRG+1] [GSoC 2019] Interface for robots.txt parsers (#3796) 
Make the robots.txt parser configurable through the new ROBOTSTXT_PARSER setting, support the Reppy and Robotexclusionrulesparser parsers, and allow implementing custom robots.txt parsers.
 2019-08-02 09:43:29 +02:00
Renne Rocha a25e09ecdd Added constrain on lxml version based on Python version 2019-07-30 23:24:41 -03:00
Mikhail Korobov 9c514b976f
Merge pull request #3450 from wRAR/tls-logging 
Log cipher, certificate and temp key info on establishing an SSL connection
 2019-07-24 01:45:40 +05:00
Eugenio Lacuesta 7e3a602d56
Suggested changes 2019-07-15 12:37:09 -03:00
Eugenio Lacuesta d5a2a70329
Fix import 2019-07-13 23:47:41 -03:00
Eugenio Lacuesta ef9a619214
More DeprecationWarnings 2019-07-13 23:36:39 -03:00
Eugenio Lacuesta eced544d64
Prevent even more DeprecationWarnings 2019-07-13 22:16:23 -03:00
Eugenio Lacuesta 40086dabb8 Prevent more DeprecationWarnings 2019-07-13 22:14:00 -03:00