1519 Commits

Author	SHA1	Message	Date
Eugenio Lacuesta	a2ae380efc	Remove unnecessary commas	2020-02-03 13:23:52 -03:00
Eugenio Lacuesta	b9e3a6201b	Merge branch 'master' into response_ip_address	2020-02-03 04:07:44 -03:00
Mikhail Korobov	22f7934fcc	Merge pull request #4269 from wRAR/asyncio-parse ... Support for async def callbacks.	2020-01-31 23:50:19 +05:00
Andrey Rakhmatullin	7687564c73	Merge remote-tracking branch 'origin/master' into asyncio-signals	2020-01-30 17:42:03 +05:00
Andrey Rakhmatullin	cc825c21de	Test returning items from an async def callback.	2020-01-30 16:17:06 +05:00
Eugenio Lacuesta	6f02a8dccb	Add source parameter to bytes_received signal	2020-01-29 15:51:25 -03:00
Eugenio Lacuesta	2c9643d38c	Test: bytes_received signal fired multiple times	2020-01-29 14:17:06 -03:00
Evgeny Dorofeev	4e56571a19	[HttpCompressionMiddleware] fix delimiter for Accept-Encoding header	2020-01-29 15:49:22 +03:00
Eugenio Lacuesta	613fd41f44	bytes_received signal: improve test performance	2020-01-27 13:25:06 -03:00
Eugenio Lacuesta	dbe20a863f	bytes_received signal: send spider argument	2020-01-27 12:28:54 -03:00
Adrián Chaves	89483ce9f7	Fix Flake8 issues	2020-01-27 12:08:56 -03:00
Eugenio Lacuesta	bda37e38bd	[Tests] bytes_received signal	2020-01-27 12:08:56 -03:00
Eugenio Lacuesta	cab449b195	Typo fix	2020-01-27 12:08:34 -03:00
Eugenio Lacuesta	e8da7e2966	Test DNS resolution using CrawlerProcess	2020-01-26 17:53:39 -03:00
Eugenio Lacuesta	80925ab845	Get server IP address for HTTP/1.1 responses	2020-01-26 16:57:04 -03:00
Mikhail Korobov	8b8df31961	Merge pull request #4227 from elacuesta/name-resolver ... Name resolver with IPv6 support	2020-01-25 05:03:13 +05:00
Mikhail Korobov	5f407cf657	Merge pull request #3961 from OmarFarrag/ftp_files#3928 ... Add FTPFileStore to FilesPipeline	2020-01-25 04:58:23 +05:00
Mikhail Korobov	7a62bd310c	Merge pull request #4126 from elacuesta/from_crawler_downloader_handlers ... Download handlers: from_crawler factory method, take crawler in __init__	2020-01-25 04:52:56 +05:00
OmarFarrag	40e0a11aa8	Fix Flake8 errors	2020-01-24 15:51:48 +02:00
OmarFarrag	fc98aa6b67	Merge branch 'master' into ftp_files#3928	2020-01-24 14:52:40 +02:00
Mikhail Korobov	f80c7776ae	Merge pull request #4008 from elacuesta/docs_request_errback ... Request: remove restriction about errback without callback	2020-01-23 23:12:44 +05:00
Mikhail Korobov	c0a7dfbc01	Merge pull request #4057 from elacuesta/response_follow_all ... Response.follow_all	2020-01-23 02:15:24 +05:00
Eugenio Lacuesta	dee420a69c	Fix name resolvers tests	2020-01-16 23:48:16 -03:00
Eugenio Lacuesta	d487498cff	Update name resolvers tests	2020-01-16 22:02:01 -03:00
Eugenio Lacuesta	90e3bd8715	[test] Name resolvers	2020-01-16 20:32:40 -03:00
Eugenio Lacuesta	6ce1ad3107	[test] Spider middleware: catch exceptions right after the spider callback	2020-01-10 04:20:37 -03:00
Andrey Rakhmatullin	a91a13b443	Support for async def callbacks.	2020-01-09 15:59:19 +05:00
Andrey Rakhmatullin	3faef2d082	Add async def support to signal handlers that already supported Deferreds.	2020-01-09 15:17:45 +05:00
Andrey Rakhmatullin	7d85984880	Use get_from_asyncio_queue in the pipeline test.	2020-01-09 14:49:16 +05:00
Andrey Rakhmatullin	bdef948aae	Mark the asyncio pipelines test as only_asyncio.	2020-01-09 14:19:02 +05:00
Andrey Rakhmatullin	bfdd552a32	Add a test for pipelines using asyncio.	2020-01-09 14:19:02 +05:00
Andrey Rakhmatullin	1f9cef787d	Add async def support to pipelines.	2020-01-09 14:19:02 +05:00
Eugenio Lacuesta	2e405d2d5c	Merge branch 'master' into response_follow_all	2020-01-05 00:33:19 -03:00
Andrey Rakhmatullin	2b9254c2bd	Add a test function that uses asyncio.Queue().	2019-12-31 17:54:41 +05:00
Andrey Rakhmatullin	16787f5bf4	Merge middleware tests back as we don't need to set the setting anymore.	2019-12-30 12:02:19 +05:00
Andrey Rakhmatullin	5cf1ac0005	Move the asyncio downloader mw test to a separate class.	2019-12-30 11:46:45 +05:00
Andrey Rakhmatullin	3603644552	Add a non-asyncio async def middleware test.	2019-12-30 11:46:45 +05:00
Andrey Rakhmatullin	21f50c795a	Add async def support to downloader middlewares.	2019-12-30 11:46:45 +05:00
Mikhail Korobov	bb991cd303	Merge pull request #4010 from scrapy/asyncio-base ... Base support for asyncio	2019-12-30 00:51:28 +05:00
Andrey Rakhmatullin	dc1ee09481	Rename ASYNCIO_ENABLED to ASYNCIO_REACTOR, change the logic accordingly.	2019-12-27 21:56:28 +05:00
Eugenio Lacuesta	c54df8253a	[test] Handle keyword args in CrawlSpider.parse	2019-12-26 15:12:19 -03:00
Eugenio Lacuesta	8d4948f6ca	[test] Override CrawlSpider.parse	2019-12-26 14:38:11 -03:00
Andrey Rakhmatullin	37ac47ff80	Fix a deprecation warning.	2019-12-26 20:46:54 +05:00
Eugenio Lacuesta	982a66f9fb	[test] Download handler: avoid passing settings if not necessary	2019-12-23 20:28:17 -03:00
nyov	b9a58798ee	change Scraper API to call internal `_parse` method ... A Spider class using internal pre-processing can have first dibs at this and then call a public `parse` method for subclass hooking.	2019-12-23 13:55:06 -03:00
Eugenio Lacuesta	a6ec89251e	Downloader handlers: crawler=None in __init__	2019-12-23 10:47:08 -03:00
Eugenio Lacuesta	8a567e98bb	Remove unnecessary __init__ methods in downloader handler tests	2019-12-23 10:36:46 -03:00
Eugenio Lacuesta	7e6387de40	Use create_instance in FTPDownloadHandler/DataURIDownloadHandler tests	2019-12-23 10:02:58 -03:00
Eugenio Lacuesta	fa21d8687a	Use create_instance in S3DownloadHandler tests	2019-12-23 10:00:25 -03:00
Eugenio Lacuesta	9e5d945ef2	Use create_instance in downloader handler tests	2019-12-23 09:55:47 -03:00
Eugenio Lacuesta	931b7e68d3	Update FileDownloadHandler test	2019-12-23 09:50:28 -03:00
Eugenio Lacuesta	310cc081d0	Merge branch 'master' into from_crawler_downloader_handlers	2019-12-21 17:24:45 -03:00
Andrey Rakhmatullin	8de80f59db	Raise an exception if ASYNCIO_ENABLED but the reactor is wrong.	2019-12-21 13:08:29 +05:00
Mikhail Korobov	5577d4d2be	Merge pull request #3869 from elacuesta/detect_return_in_generator_callbacks ... [MRG+1][Py3] Check for 'return' with arguments inside generators	2019-12-20 22:46:46 +05:00
Andrey Rakhmatullin	e342de5038	Remove a stray newline.	2019-12-20 19:37:50 +05:00
Andrey Rakhmatullin	20289be810	Merge remote-tracking branch 'origin/master' into asyncio-base	2019-12-20 19:31:20 +05:00
Mikhail Korobov	fb3fb17c28	Merge pull request #4045 from Gallaecio/documentation-coverage ... Deprecate FilteringLinkExtractor direct usage; add API docs for scrapy.linkextractors	2019-12-20 02:01:45 +05:00
Mikhail Korobov	3bcb14fe4d	Merge pull request #4164 from Gallaecio/speed-up-tests-2 ... Improve the performance of the DOWNLOAD_DELAY test	2019-12-18 23:56:56 +05:00
Marc Hernandez Cabot	a5de2c64e6	fix W291, W292, W293 (whitespaces)	2019-12-18 16:33:46 +01:00
elacuesta	916382e109	Add errback parameter to scrapy.spiders.crawl.Rule (#4000) ... * Add errback parameter to scrapy.spiders.crawl.Rule * CrawlSpider: optimize by reducing iterations * [test] Rule.errback * [doc] Rule.errback * [doc] Use autoclass in docs/topics/spiders.rst Co-Authored-By: Adrián Chaves <adrian@chaves.io> * Rule.process_links takes a list * Fix aesthetic issue reported by Flake8	2019-12-18 20:05:33 +05:00
Andrey Rakhmatullin	7ccb169a27	Split a long test in test_engine.py into three.	2019-12-18 19:41:16 +05:00
Mikhail Korobov	e90f276903	Merge pull request #4198 from wRAR/deprecate-noconnect ... Deprecate the HTTPS proxy noconnect mode.	2019-12-18 16:11:24 +05:00
Adrián Chaves	1f689b50c6	Merge remote-tracking branch 'origin/documentation-coverage' into documentation-coverage	2019-12-18 12:09:34 +01:00
Adrián Chaves	174769a3f0	Use a better name for the LxmlLinkExtractor subclassing test	2019-12-18 12:09:03 +01:00
Andrey Rakhmatullin	ac302c3f61	Fix a flake8 problem.	2019-12-18 15:43:05 +05:00
Andrey Rakhmatullin	bb2ff13e4c	Skip Http10ProxyTestCase.test_download_with_proxy_https_noconnect	2019-12-18 15:39:08 +05:00
Andrey Rakhmatullin	2d92a39003	Restore test_download_with_proxy_https_noconnect, check for a warning there.	2019-12-18 12:07:08 +05:00
Adrián Chaves	607815dfc2	Merge branch 'master' into documentation-coverage	2019-12-17 16:54:27 +01:00
Andrey Rahmatullin	bb3f164280	Merge pull request #4236 from wRAR/pipeline-tests ... Add simple tests for pipelines.	2019-12-17 19:38:06 +05:00
Marc Hernández	63cf5c75c8	Fix E502: backslash is redundant between brackets (#4238)	2019-12-17 13:53:15 +01:00
Andrey Rakhmatullin	5980d3bbff	Add simple tests for pipelines.	2019-12-16 22:41:28 +05:00
Eugenio Lacuesta	451e7a616e	Scan callbacks/errbacks for return statements with values different than None	2019-12-16 11:30:40 -03:00
Adrián Chaves	a4ef9750f9	Fix Flake8-reported issues	2019-12-13 14:32:06 +01:00
Adrián Chaves	b5c4c2cae8	Keep 2 spaces between code and inline comments (#4195)	2019-12-13 14:20:48 +01:00
Andrey Rakhmatullin	bfb78b8dea	Add CrawlerProcess tests for ASYNCIO_ENABLED.	2019-12-13 18:12:07 +05:00
Andrey Rakhmatullin	3560123090	Rename ASYNCIO_SUPPORT to ASYNCIO_ENABLED.	2019-12-13 18:07:47 +05:00
Andrey Rakhmatullin	352ddbb336	Merge remote-tracking branch 'origin/master' into asyncio-base	2019-12-13 18:07:23 +05:00
Andrey Rahmatullin	02cdc53fb8	Add a test for a CrawlerProcess script. (#4218) ... * Add a test for a CrawlerProcess script. * Add tests/CrawlerProcess to collect_ignore. * Remove an extra line. * Fix/improve conftest.py.	2019-12-13 18:04:05 +05:00
Eugenio Lacuesta	21a2f8c6f8	Merge branch 'master' into response_follow_all	2019-12-10 17:30:59 -03:00
Adrián Chaves	1fc2b140c1	Merge branch 'master' into documentation-coverage	2019-12-05 14:43:36 +01:00
Andrey Rahmatullin	076f0764b7	Merge pull request #4121 from scrapy/remove-six-code ... Remove six-related code and __future__ imports	2019-12-05 18:20:14 +05:00
Mikhail Korobov	250da28952	Merge pull request #4170 from mabelvj/4133-handle-start_url ... Raise error when start_url found instead of start_urls.	2019-12-05 17:47:03 +05:00
Andrey Rakhmatullin	ed34ce14c0	Add the ASYNCIO_SUPPORT setting, reshuffle other logic accordingly.	2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin	cc19ab5439	Add tests that check asyncio support.	2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin	b04b541372	Install the asyncio reactor only in scrapy.cmdline.	2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin	8d8fbddbde	Switch to the released version of pytest-twisted.	2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin	9777639533	Run tests using the asyncio reactor.	2019-12-04 21:32:56 +05:00
Andrey Rakhmatullin	1b35260625	Add a test for downloader middlewares using Deferreds.	2019-12-04 21:32:56 +05:00
Eugenio Lacuesta	62778cf23f	Request: remove restriction about errback without callback	2019-12-04 08:53:38 -03:00
Eugenio Lacuesta	2a9f5a0aef	Skip invalid links when passing SelectorLists to Response.follow_all	2019-12-03 16:01:42 -03:00
Eugenio Lacuesta	e43f37fff3	Pass args/kwargs in S3DownloadHandler.from_crawler, update tests	2019-12-02 17:36:31 -03:00
Eugenio Lacuesta	3d77f74e40	Download handlers: from_crawler factory method, take crawler instead of settings in __init__	2019-12-02 17:36:31 -03:00
Andrey Rakhmatullin	63546cbf3e	Deprecate the HTTPS proxy noconnect mode.	2019-11-27 22:42:52 +05:00
Adrián Chaves	6d9ed6146d	Merge branch 'master' into remove-six-code	2019-11-25 10:34:21 +01:00
Eugenio Lacuesta	03af8885ff	LogFormatter.download_error	2019-11-23 20:02:44 -03:00
Eugenio Lacuesta	4756e7c587	LogFormatter.spider_error	2019-11-23 19:33:29 -03:00
Eugenio Lacuesta	facb926542	Remove quotes from item_error message	2019-11-23 19:16:41 -03:00
Eugenio Lacuesta	7a7d13b112	Rename LogFormatter.error to item_error	2019-11-23 19:12:03 -03:00
Eugenio Lacuesta	40b5cfc0a4	Item loaders: allow single-argument processors (unbound methods)	2019-11-22 20:47:22 -03:00
Eugenio Lacuesta	6fa8f06b64	Merge remote-tracking branch 'upstream/master' into response_follow_all	2019-11-22 13:05:06 -03:00
Eugenio Lacuesta	5bab3c0261	Merge remote-tracking branch 'upstream/master' into remove-six-code	2019-11-22 12:12:29 -03:00
Mikhail Korobov	16e0636dcf	Merge pull request #4186 from Gallaecio/lgtm ... Remove unused imports	2019-11-22 12:28:29 +05:00
Mabel Villalba	070b3a4e84	Merge branch 'master' into 4133-handle-start_url	2019-11-21 17:10:31 +01:00
Mabel Villalba	1718e450ef	[start_url] Fixes #4133: Raise AttributeError error when empty 'start_urls' and 'start_url' found. Added test.	2019-11-21 17:06:47 +01:00
Mikhail Korobov	d2b73b8e21	Merge pull request #4114 from scrapy/remove-py2-tests ... Remove py2 tests	2019-11-21 18:22:10 +05:00
Adrián Chaves	a2bf340bab	Remove unused imports	2019-11-21 14:21:17 +01:00
Eugenio Lacuesta	714e0d3ac9	Merge remote-tracking branch 'upstream/master' into remove-six-code	2019-11-20 12:41:38 -03:00
Mikhail Korobov	65e2fb7d89	Merge pull request #4153 from Gallaecio/lgtm ... MutableChain: return self from __iter__	2019-11-20 19:46:20 +05:00
Mikhail Korobov	e96ca89a1d	Merge pull request #4174 from Gallaecio/flake8 ... Add missing blank lines between functions and classes	2019-11-20 19:34:04 +05:00
Eugenio Lacuesta	6781d2f5b2	Update sample file references	2019-11-20 09:58:25 -03:00
Eugenio Lacuesta	48223c107a	Merge remote-tracking branch 'upstream/master' into response_follow_all	2019-11-20 09:41:42 -03:00
Eugenio Lacuesta	b602c61e1c	[Test] Rename outdated sample files	2019-11-20 09:38:54 -03:00
Eugenio Lacuesta	6f6426559d	Merge remote-tracking branch 'upstream/remove-py2-tests' into remove-six-code	2019-11-19 11:04:46 -03:00
Eugenio Lacuesta	05785c1c17	Merge remote-tracking branch 'upstream/master' into remove-six-code	2019-11-19 11:01:34 -03:00
Andrey Rahmatullin	78ad01632f	Fix flake8 problems in PR #3989 (#4176)	2019-11-19 10:43:30 +01:00
Andrey Rakhmatullin	beb7d80d6a	Add a comment about the noconnect tests.	2019-11-19 13:52:49 +05:00
Andrey Rakhmatullin	922a66cf07	Fix or ignore flake8 problems.	2019-11-19 13:52:49 +05:00
Andrey Rakhmatullin	a7b640991d	Rename tests/py3-ignores.txt to tests/ignores.txt.	2019-11-19 13:52:49 +05:00
Andrey Rakhmatullin	8b730a3670	Use self.proc.communicate() after killing mitmdump.	2019-11-19 13:52:49 +05:00
Andrey Rakhmatullin	5970d00eb9	Only xfail test_https_connect_tunnel_error on 3.6+.	2019-11-19 13:52:49 +05:00
Andrey Rakhmatullin	5080180c75	Improve the test_https_tunnel_without_leak_proxy_authorization_header change.	2019-11-19 13:52:49 +05:00
Andrey Rakhmatullin	c4ef950efd	Use an older mitmproxy for py3.5.	2019-11-19 13:52:49 +05:00
Andrey Rakhmatullin	3ec6960732	Fix test_proxy_connect.py for py3.5.	2019-11-19 13:52:49 +05:00
Andrey Rakhmatullin	c327ad9ba6	Remove an unused six import.	2019-11-19 13:52:49 +05:00
Andrey Rakhmatullin	cbb6d0c6a7	Mark failing proxy tests.	2019-11-19 13:52:49 +05:00
Andrey Rakhmatullin	f066257e95	Restore tests/test_proxy_connect.py and update it to modern mitmproxy.	2019-11-19 13:52:49 +05:00
Andrey Rakhmatullin	e18014d84d	Remove Python 2-only tests.	2019-11-19 13:52:49 +05:00
Andrey Rahmatullin	25cd7ac7ab	Merge pull request #4094 from victor-torres/invalid-url ... Improve URL schema validation on scrapy.Request initialization	2019-11-19 13:50:08 +05:00
Andrey Rahmatullin	3408b757c1	Merge pull request #3989 from elacuesta/logformatter_error_method ... LogFormatter improvements	2019-11-19 13:44:43 +05:00
Adrián Chaves	f261cf65e9	Add missing blank lines between functions and classes ... Also fixed 2 unrelated Flake8 issues	2019-11-18 17:16:09 +01:00
Adrián Chaves	99d8b05a0b	Deprecate scrapy.utils.python.MutableChain.next	2019-11-18 10:58:47 +01:00
Marc Hernandez Cabot	0e252f5a13	fix E711 and E713	2019-11-15 19:12:43 +01:00
Mikhail Korobov	550d8046fd	Merge pull request #4157 from Gallaecio/flake8 ... Add missing whitespace after ‘,’, ‘;’ or ‘:’	2019-11-14 23:42:06 +05:00
Andrey Rakhmatullin	3b2289ad01	Rename test_non_str_url_py2 to test_bytes_url.	2019-11-14 22:53:28 +05:00
Andrey Rakhmatullin	fe3a121f13	Use kwargs when calling get_func_args.	2019-11-14 22:50:53 +05:00
Adrián Chaves	058bdda0af	Improve the performance of the DOWNLOAD_DELAY test	2019-11-14 16:53:20 +01:00
Andrey Rakhmatullin	e291460db6	Fix flake8-detected errors.	2019-11-14 15:24:37 +05:00
Adrián Chaves	1a4a77d49f	Remove Python 2 check from MutableChainTest	2019-11-14 10:24:31 +01:00
Andrey Rakhmatullin	87c23ba22d	Remove Py2-only code that checks sys.version_info.	2019-11-14 13:45:39 +05:00
Andrey Rakhmatullin	a138fb05d4	Replace to_native_str calls with to_unicode.	2019-11-14 13:45:39 +05:00
Andrey Rakhmatullin	92ffd2f249	Simplify some more imports.	2019-11-14 13:45:39 +05:00
Andrey Rakhmatullin	85e79ae792	Remove cStringIO imports.	2019-11-14 13:45:39 +05:00
Andrey Rakhmatullin	1d7c8cb0b1	Remove six.PY2 and six.PY3 conditionals.	2019-11-14 13:45:37 +05:00
Adrián Chaves	33ef24c757	Add missing whitespace after ‘,’, ‘;’ or ‘:’	2019-11-13 10:52:05 +01:00
Andrey Rahmatullin	c911e80209	Merge pull request #4088 from ammarnajjar/4086-constructor-initializer-docs ... docs: use __init__ method instead of constructor	2019-11-12 21:48:16 +05:00
Andrey Rahmatullin	c91390558d	Merge pull request #4074 from purvaudai/master ... Added Pathlib.Path support: Issue #3731	2019-11-12 21:43:28 +05:00
purvaudai	4b8b0345e5	Mades Changes as per review	2019-11-12 18:17:15 +05:30
Andrey Rahmatullin	93385e647a	Merge pull request #4123 from elacuesta/utils-local-cache-limit ... Fix LocalCache limit issue, add tests	2019-11-12 16:05:22 +05:00
purvaudai	50eaabe1fc	Added Test	2019-11-11 20:00:26 +05:30
purvaudai	f39ff49458	Added Test	2019-11-11 18:54:21 +05:30
purvaudai	970c3be160	Added Test	2019-11-11 18:34:15 +05:30
purvaudai	9e6e2dde2b	Adding test	2019-11-11 16:10:37 +05:30
purvaudai	0042c389eb	Adding test	2019-11-11 15:57:58 +05:30
purvaudai	084a1cda6d	Adding test	2019-11-11 15:41:00 +05:30
Eugenio Lacuesta	07fa768d33	Merge branch 'remove-six-code' of github.com:scrapy/scrapy into remove-six-code	2019-11-08 22:17:12 -03:00
Eugenio Lacuesta	b6bbb28197	PEP8 adjustments	2019-11-08 22:13:03 -03:00
Eugenio Lacuesta	342e3b5bd5	Merge remote-tracking branch 'upstream/master' into remove-six-code	2019-11-08 12:29:15 -03:00
Adrián Chaves	1df5755699	Set the bases for testing code examples from the documentation	2019-11-08 16:00:10 +01:00
elacuesta	44f19df311	[test] Update mitmproxy version ... Co-Authored-By: Adrián Chaves <adrian@chaves.io>	2019-11-08 11:32:50 -03:00
Marc Hernández	c377c14e32	Fix W391 Blank line at end of file (#4137)	2019-11-07 17:47:35 +01:00
elacuesta	fe31695ba0	Remove unused import (urllib.parse.unquote) ... Co-Authored-By: Adrián Chaves <adrian@chaves.io>	2019-11-05 15:36:19 -03:00
Eugenio Lacuesta	613c66a034	Do not override built-in max function	2019-11-05 09:45:51 -03:00
elacuesta	fed9fbe62d	Update tests/test_utils_datatypes.py ... Co-Authored-By: Adrián Chaves <adrian@chaves.io>	2019-11-04 15:34:27 -03:00
Eugenio Lacuesta	439a3e59b8	Fix scrapy.utils.datatypes.LocalCache limit issue	2019-11-04 10:35:58 -03:00
Eugenio Lacuesta	00b793dc59	Remove elluding six occurrences	2019-11-03 12:26:38 -03:00
Eugenio Lacuesta	5797aefd4c	Remove six.assertCountEqual	2019-11-03 12:18:35 -03:00
Eugenio Lacuesta	586b25d27e	Remove six types	2019-11-03 12:10:37 -03:00
Eugenio Lacuesta	7f3cb05d8e	Remove metaclass-related six code	2019-11-03 12:03:02 -03:00
Eugenio Lacuesta	cca1959c07	Merge remote-tracking branch 'upstream/remove-py2-tests' into remove-six-code	2019-11-03 01:20:08 -03:00
Eugenio Lacuesta	d72444b9c8	Remove more six imports	2019-11-03 01:11:28 -03:00
Eugenio Lacuesta	eaeaa40b99	Remove six.PY* checks	2019-11-03 01:08:08 -03:00
Eugenio Lacuesta	5d8abdde59	Remove six.text_type from tests	2019-11-03 01:01:23 -03:00
Eugenio Lacuesta	54a786b102	Remove six imports	2019-11-03 00:58:47 -03:00
Eugenio Lacuesta	1aba513693	Remove six.iter* occurrences	2019-11-03 00:26:44 -03:00
Eugenio Lacuesta	5ab0f189ce	Remove six.moves occurrences from tests	2019-11-03 00:01:09 -03:00
Eugenio Lacuesta	c0bfaef37a	Remove __future__ imports from tests	2019-11-02 23:27:04 -03:00
Andrey Rakhmatullin	350aa67c3d	Rename tests/py3-ignores.txt to tests/ignores.txt.	2019-11-01 19:52:57 +05:00
Andrey Rakhmatullin	4b0cdf7f3e	Use self.proc.communicate() after killing mitmdump.	2019-11-01 19:51:52 +05:00
Andrey Rakhmatullin	3c9963ab04	Only xfail test_https_connect_tunnel_error on 3.6+.	2019-11-01 19:47:12 +05:00
Andrey Rakhmatullin	e0c5c72496	Improve the test_https_tunnel_without_leak_proxy_authorization_header change.	2019-11-01 19:46:19 +05:00
Andrey Rakhmatullin	5eb01b617d	Use an older mitmproxy for py3.5.	2019-10-31 23:21:30 +05:00
Andrey Rakhmatullin	a5eb59b92d	Fix test_proxy_connect.py for py3.5.	2019-10-31 23:21:14 +05:00
Adrián Chaves	f02c3d1dcf	Use communicate() instead of wait() after killing the mock server (#4095)	2019-10-31 13:31:33 +01:00
Andrey Rakhmatullin	7299e91b1f	Remove Py2-only code that checks sys.version_info.	2019-10-31 15:22:53 +05:00
Andrey Rakhmatullin	397e883556	Replace to_native_str calls with to_unicode.	2019-10-31 15:22:53 +05:00
Andrey Rakhmatullin	75b1d051d9	Simplify some more imports.	2019-10-31 15:22:45 +05:00
Andrey Rakhmatullin	3ac4b430ae	Remove an unused six import.	2019-10-31 15:20:28 +05:00
Andrey Rakhmatullin	cea2f5e244	Remove cStringIO imports.	2019-10-31 15:07:54 +05:00
Andrey Rakhmatullin	bbd9f4be90	Remove six.PY2 and six.PY3 conditionals.	2019-10-31 15:07:10 +05:00
Andrey Rakhmatullin	186f9d88ac	Fix the skip message for test_download_gzip_response.	2019-10-31 15:03:44 +05:00
Andrey Rakhmatullin	439e37fc7b	Mark failing proxy tests.	2019-10-31 15:03:44 +05:00
Andrey Rakhmatullin	b0d6f4917d	Restore tests/test_proxy_connect.py and update it to modern mitmproxy.	2019-10-31 15:03:44 +05:00
Andrey Rakhmatullin	b44bd6f825	Remove Python 2-only tests.	2019-10-31 15:03:09 +05:00
Adrián Chaves	15c55d0c1d	Remove LevelDB support (#4112)	2019-10-31 10:47:29 +01:00
Andrey Rahmatullin	229e722a03	Initial Python 2 removal (#4091)	2019-10-31 10:46:02 +01:00
Benjamin Ooghe-Tabanou	6d6da78eda	Add a keep_fragments parameter to the request_fingerprint function (#4104)	2019-10-30 09:13:36 +01:00
Amardeep Bhowmick	66cbceeb0a	Fix redirection error when the Location header value starts with 3 slashes (#4042)	2019-10-30 09:09:12 +01:00
Andrey Rahmatullin	18b808b2e9	Merge pull request #4092 from further-reading/master ... Add Python 3.8 official support	2019-10-29 16:30:58 +05:00
Roy	93e3dc1b82	[test_downloadermiddleware_httpcache.py] Cleaning text ... https://github.com/scrapy/scrapy/issues/4085	2019-10-28 16:12:03 +00:00
Roy	b73d217de5	[test_downloadermiddleware_httpcache.py] Fixing pytest mark behaviour ... https://github.com/scrapy/scrapy/issues/4085	2019-10-28 12:55:54 +00:00
Roy	c51fb959e2	[test_downloadermiddleware_httpcache] Fixing pytest skip behaviour ... https://github.com/scrapy/scrapy/issues/4085	2019-10-28 12:36:54 +00:00
Roy	4432136fff	[test_downloadermiddleware_httpcache] Fixing pytest skip behaviour ... https://github.com/scrapy/scrapy/issues/4085	2019-10-28 12:22:21 +00:00
Roy	16bb3ac20d	[test_downloadermiddleware_httpcache] Using skipif approach ... https://github.com/scrapy/scrapy/issues/4085	2019-10-28 11:24:09 +00:00
elacuesta	7731814cc2	ItemLoader: improve handling of initial item (#4036)	2019-10-28 10:53:53 +01:00
Roy	20ea912513	[test_downloadermiddleware_httpcache] Making xfails more informative ... https://github.com/scrapy/scrapy/issues/4085	2019-10-27 18:52:01 +00:00
Roy	70b2854590	[test_downloadermiddleware_httpcache] Making xfails more informative ... https://github.com/scrapy/scrapy/issues/4085	2019-10-27 18:51:26 +00:00
Roy	b3df0a8415	[test_downloadermiddleware_httpcache] Adding xfails to impacted tests following hack fix ... https://github.com/scrapy/scrapy/issues/4085	2019-10-27 18:28:47 +00:00
Roy	11942c436c	[test_downloadermiddleware_httpcache] Trying hack to handle systemerror whjen importing leveldb ... https://github.com/scrapy/scrapy/issues/4085	2019-10-27 18:02:13 +00:00
Roy	deacd34c8d	[test_downloadermiddleware_httpcache] Attempting to add xfail for leveldb related tests ... https://github.com/scrapy/scrapy/issues/4085	2019-10-27 17:39:47 +00:00
Roy Healy	4068797558	Update test_downloadermiddleware_httpcache.py ... Adding xfail denoting that leveldb is not supported in 3.8	2019-10-27 17:02:17 +00:00
Victor Torres	f701f5b0db	fix #2552 by improving request schema check on its initialization	2019-10-22 10:48:02 -03:00
Ammar Najjar	7a84a4bdba	docs: use "constructor" for from_settings() & rom_crawler() factory methods ... Issue #4086	2019-10-22 15:31:34 +02:00
Adrián Chaves	1d5c270ce8	Fix dangling file descriptor in FeedExporter when FEED_STORE_EMPTY is False (#4023)	2019-10-22 15:12:52 +02:00
purvaudai	4184bac068	Added Pathlib.Path test	2019-10-22 16:57:14 +05:30
purvaudai	85f56a92f0	Added Pathlib.Path test	2019-10-22 16:43:17 +05:30
purvaudai	7031e3a124	Added Pathlib.Path test	2019-10-22 16:31:14 +05:30
purvaudai	cd4c211f4b	Added Pathlib.Path test	2019-10-22 15:38:06 +05:30
purvaudai	a776554282	Added Pathlib.Path test	2019-10-22 15:31:55 +05:30
purvaudai	ad96d6ef59	Added Pathlib.Path test correctly	2019-10-22 14:53:59 +05:30
purvaudai	2ee38e8ddb	Added Pathlib.Path test	2019-10-22 14:43:47 +05:30
Ammar Najjar	da8cd9448d	docs: always surround __init__ with `` in docs ... Issue #4086	2019-10-21 19:56:58 +02:00
Ammar Najjar	68a7d05ed8	docs: use __init__ method instead of constructor ... Issue #4086	2019-10-21 15:42:24 +02:00
Eugenio Lacuesta	e1fa1fd8ad	TextResponse.follow_all: skip invalid links	2019-10-10 00:37:30 -03:00
Eugenio Lacuesta	5f168cd459	Response.follow_all	2019-10-08 16:32:56 -03:00
Eugenio Lacuesta	e0fabab5cc	Fix TypeError when using DummyStatsCollector	2019-10-01 18:03:22 -03:00
Eugenio Lacuesta	07a31b13db	Update LogFormatter tests	2019-10-01 17:55:57 -03:00
Adrián Chaves	7f4f98fd38	Provide complete API documentation coverage of scrapy.linkextractors	2019-09-30 18:22:28 +02:00
s-sanjay	2c14692e60	remove .keys() to avoid creating a tmp list/keyview obj (#4031) ... Also add --verbose and --nolinks for code coverage	2019-09-27 09:56:43 +02:00
Mikhail Korobov	31c631f9ee	Merge pull request #3999 from Gallaecio/documentation-coverage ... Provide complete API documentation coverage of scrapy.item	2019-09-25 16:13:13 +01:00
Adrián Chaves	1236e9e81e	Provide complete API documentation coverage of scrapy.item	2019-09-25 13:24:56 +02:00
Mikhail Korobov	1364e92007	Merge pull request #4015 from elacuesta/remove_deprecated_xlib ... Remove deprecated xlib module	2019-09-20 15:28:07 +01:00
Mikhail Korobov	17c0d72d04	Merge pull request #4016 from elacuesta/rule_default_link_extractor ... Crawling rules: make link extractors optional	2019-09-20 15:27:17 +01:00
Eugenio Lacuesta	27436cbbc9	[test] LogFormatter.error	2019-09-16 12:12:10 -03:00
watsta	0b52fa6ca9	LogFormatter: Add the ability to skip log messages (#3987)	2019-09-16 14:12:04 +02:00
OmarFarrag	13735bcf34	Disallow media extensions unregistered with IANA (#3954) ... Co-Authored-By: s-sanjay <sanjay537@gmail.com>	2019-09-16 14:04:06 +02:00
Eugenio Lacuesta	21ad8e20b9	Crawling rules: make link extractors optional	2019-09-13 17:32:42 -03:00
Eugenio Lacuesta	c5612f387b	Remove deprecated xlib module	2019-09-13 14:32:05 -03:00
Mikhail Korobov	534de7395d	Merge pull request #3988 from elacuesta/contracts_cb_kwargs ... CallbackKeywordArgumentsContract	2019-09-08 04:23:16 +05:00
OmarFarrag	0e8770a2f4	test for files pipeline ftp store	2019-09-06 15:47:57 +02:00
Eugenio Lacuesta	b92b114633	[test] cb_kwargs contract	2019-08-31 02:44:09 -03:00
Marc Hernández	ace2df3d14	Fix JSONRequest naming (#3982)	2019-08-30 11:03:44 +02:00
Júlio César Batista	b84f99ff5d	Merge	2019-08-29 11:11:56 -03:00
Júlio César Batista	110bc92e6b	Fix default value of FEED_STORAGE_GCS_ACL	2019-08-29 11:10:00 -03:00
Mikhail Korobov	ede91478e5	Merge pull request #3966 from anubhavp28/robotstxt_useragent ... Adds ROBOTSTXT_USER_AGENT setting	2019-08-28 22:00:17 +05:00
Anubhav Patel	3a7b949d6d	Adds integration with Protego robots.txt parser (#3935)	2019-08-27 09:41:31 +02:00
Anubhav Patel	00fe05e536	adds ROBOTSTXT_USER_AGENT setting	2019-08-19 09:24:16 +05:30
Andrey Rakhmatullin	aaa5229e5d	Fixes and improvements for DOWNLOADER_CLIENT_TLS_CIPHERS.	2019-08-13 16:56:26 +05:00
Andrey Rakhmatullin	9a8edf2bf1	Tests for setting SSL ciphers.	2019-08-13 16:53:19 +05:00
Mikhail Korobov	73d1b4b748	Merge pull request #3939 from Gallaecio/improve-scrapes-contract ... Report all missing fields when a scrapes contract fails	2019-08-08 13:49:09 +05:00
Mikhail Korobov	b4556d6508	Merge pull request #3941 from starrify/ftp-storage-uri-unquote ... [MRG+1] Added: Properly handling quoted passwords in FEED_URI for FTP	2019-08-08 13:38:17 +05:00
Adrián Chaves	9119798a5c	Add test coverage for contract failures involving multiple missing fields	2019-08-08 09:52:03 +02:00
Marc Hernández	d76b6944c9	Create Request from curl command (#3862)	2019-08-08 09:43:42 +02:00
elacuesta	5dbeece8da	[MRG+1] Drop py34 support - Update CI envs (#3892) ... * Drop py34 support * Travis experiments * More Travis experiments * Bump Twisted version for py35+ (stretch) * Remove Debian build * Remove pinned lxml for Py34 * Fix merge error * Remove unused tox env * Add environment with pinned versions for py36 * Bump minimum Twisted version in py27; Envs with pinned versions for py27 and py35 * Add botocore as extra dep for py27 tests * Update requirements-py2.txt * Add botocore and Pillow as extra dependencies	2019-08-07 12:36:52 +05:00
Pengyu Chen	7b755a41a1	Added: Properly handling quoted passwords in FEED_URI for FTP	2019-08-06 15:19:57 +01:00
Anubhav Patel	8e813953bd	[MRG+1] [GSoC 2019] Interface for robots.txt parsers (#3796) ... Make the robots.txt parser configurable through the new ROBOTSTXT_PARSER setting, support the Reppy and Robotexclusionrulesparser parsers, and allow implementing custom robots.txt parsers.	2019-08-02 09:43:29 +02:00
Renne Rocha	a25e09ecdd	Added constrain on lxml version based on Python version	2019-07-30 23:24:41 -03:00
Mikhail Korobov	9c514b976f	Merge pull request #3450 from wRAR/tls-logging ... Log cipher, certificate and temp key info on establishing an SSL connection	2019-07-24 01:45:40 +05:00
Eugenio Lacuesta	7e3a602d56	Suggested changes	2019-07-15 12:37:09 -03:00
Eugenio Lacuesta	d5a2a70329	Fix import	2019-07-13 23:47:41 -03:00
Eugenio Lacuesta	ef9a619214	More DeprecationWarnings	2019-07-13 23:36:39 -03:00
Eugenio Lacuesta	eced544d64	Prevent even more DeprecationWarnings	2019-07-13 22:16:23 -03:00
Eugenio Lacuesta	40086dabb8	Prevent more DeprecationWarnings	2019-07-13 22:14:00 -03:00
Eugenio Lacuesta	92d624c161	[Tests] Prevent more TestCase DeprecationWarnings ... * assertRegexpMatches -> assertRegex	2019-07-13 20:55:45 -03:00
Eugenio Lacuesta	b714a372e2	[Tests] Prevent more DeprecationWarnings ... * assertRegexpMatches -> assertRegex * invalid escape sequence \[	2019-07-13 20:23:15 -03:00
Eugenio Lacuesta	09e27d2d2e	[Tests] Monkey patch unittest.TestCase to prevent DeprecationWarning(s)	2019-07-13 19:26:59 -03:00
Andrey Rakhmatullin	a96a07bc76	Add a test for DOWNLOADER_CLIENT_TLS_VERBOSE_LOGGING.	2019-07-12 18:44:45 +05:00
Luiz Francisco Rodrigues da Silva	df68c4b9b1	Add a setting to use active mode in FTPFeedStorage (#3829) ... Add a setting to use active mode in FTPFeedStorage FTP servers can be configured in active and passive mode, by default Python and Scrapy use passive mode and there was no way to use active. This commit adds a setting FEED_STORAGE_FTP_ACTIVE to allow a feed to be exported to a FTP server configured in active mode.	2019-07-10 14:25:52 +02:00
nyov	5442c2d3c3	Updating S3FeedStorage instancing without AWS key.	2019-07-06 13:41:56 +00:00
nyov	7a398b7086	Deprecation removals for Scrapy 1.7 ... Removing deprecations of 2015 and prior (pre-1.1)	2019-07-06 13:41:56 +00:00
Daniel Graña	3d8f075b0a	Merge pull request #3840 from mabelvj/itemloader-errors ... [MRG+1] Itemloader errors #3836	2019-07-05 08:55:43 -03:00
Mikhail Korobov	9aec7856b0	Merge pull request #3794 from csalazar/whitelist-form-methods-in-fromresponse ... [MRG+1] Fix form methods in FormRequest.from_response (#3777)	2019-07-02 20:08:14 +05:00
Eugenio Lacuesta	312e57382c	Request.cb_kwargs: update in spider middleware	2019-06-26 12:48:00 -03:00
Eugenio Lacuesta	d4d68cf321	Request.cb_kwargs: update in downloader middleware	2019-06-26 12:31:41 -03:00
Eugenio Lacuesta	1f9f41b850	Move request.cb_kwargs tests to their own test file	2019-06-26 12:31:12 -03:00
Eugenio Lacuesta	428309ba1a	Merge remote-tracking branch 'origin/master' into callback_kwargs	2019-06-26 11:03:31 -03:00
Mikhail Korobov	c81d120bde	Merge pull request #3768 from Gallaecio/meta-ignore-tags ... Implement the METAREFRESH_IGNORE_TAGS setting	2019-06-25 23:25:58 +03:00
Mabel Villalba	b1e348b281	[itemloader-errors] updated Compose and MapCompose messages and added tests ... #3836	2019-06-25 17:11:14 +02:00
Mabel Villalba	a753ea7e16	[itemloader-errors] added test for Compose and MapCompose ... #3836	2019-06-25 14:35:49 +02:00
Mabel Villalba	99eb03a810	[itemloader-errors] adapted compose test for the new error being returned by compose ... #3836	2019-06-25 14:24:27 +02:00
Victor Torres	f4f2b1695c	Fix a memory leak on the Media Pipeline (Files and Images) (#3813) ... We're storing exceptions captured by Twisted on the media pipeline cache, but we're also using the defer.returnValue method with our own methods decorated with @defer.inlineCallbacks. The defer.returnValue method passes returned values forward by throwing a defer._DefGen_Return exception, which in its turn extends the BaseException class and is captured by Twisted. This way, the latest exception stored in the Failure's object may also have an HtmlResponse object in its __context__ attribute. As the Response object also keeps track of the Request object that has originated it, you could figure it out how many RAM we're wasting here. This could easily lead to a Memory Leak problem when running spiders with Media Pipeline enabled and a particular Request set that tends to raise a significant number of exceptions. Example triggers: - media requests with 404 status responses - user land exceptins coming from custom middlewares - etc.	2019-06-24 12:38:05 +02:00
Daniel Graña	b53ff59a22	Merge pull request #3790 from andrewbaxter/master ... [MRG+1] Account for mangling when serializing requests with private callbacks	2019-06-17 12:33:21 -03:00
Daniel Graña	31cdb940ff	Merge pull request #3833 from anubhavp28/remove_unused_var ... removes unused var	2019-06-17 11:50:49 -03:00
Anubhav Patel	b8900ec6a6	removes unused var	2019-06-17 00:06:44 +05:30
Daniel Graña	cdfb20aee8	Merge pull request #3739 from Matthijsy/feature/scrapy_check_env ... [MRG+1] Add SCRAPY_CHECK environment variable	2019-06-13 19:43:01 -03:00
sortafreel	cdeccac6d6	Linting (return previous indentation).	2019-06-11 17:38:06 +03:00
sortafreel	7dad2f7b13	Add more tests.	2019-06-11 07:43:03 +03:00
sortafreel	a1bca6a8e7	Add tests.	2019-06-11 07:36:29 +03:00
Claudio Salazar	0c50879568	Change behavior to use method GET when there are unknown methods ... in the form	2019-06-06 22:10:59 +02:00
Andrew Baxter	6af1dc89aa	Fix mangling test	2019-06-06 04:25:19 +09:00
Andrew Baxter	3dd3e8c298	Restrict different class mangling tests to Py 3+	2019-06-05 23:49:54 +09:00
Andrew Baxter	9c81721c40	Add tests for private method name mangling	2019-06-05 23:43:56 +09:00
Andrew Baxter	bcad8947e8	Support inherited private method names	2019-06-03 20:41:02 +09:00
Andrew Baxter	9af91a26b0	Replace regex usage	2019-05-28 01:40:26 +09:00
Andrew Baxter	72b7d3e90a	Make the regex align to the spec better; add unit tests for name variations	2019-05-27 23:30:23 +09:00
Claudio Salazar	461682fc3d	Whitelist form methods in FormRequest.from_response method	2019-05-25 11:01:19 +02:00
Andrew Baxter	e667ca7682	Account for mangling when serializing requests with private callbacks	2019-05-24 21:45:53 +09:00
Adrián Chaves	611249bb7f	Implement the METAREFRESH_IGNORE_TAGS setting	2019-05-08 12:52:29 +02:00
Matthijs Vos	f6485e6697	Restore alphabetic order and two lines between import and code	2019-05-03 13:55:18 +02:00
Matthijs Vos	6d52708579	Add reset case	2019-04-18 15:19:23 +02:00
Matthijs Vos	2973998947	Add set_environ test	2019-04-18 14:50:02 +02:00
Mikhail Korobov	5b667b6156	Merge pull request #3707 from Gallaecio/pytest-xdist ... Use pytest-xdist	2019-04-17 13:20:37 +05:00
Adrián Chaves	5a6fb3daa6	Use pytest-xdist	2019-04-17 09:02:34 +02:00
Mikhail Korobov	9280185054	Merge pull request #3520 from whalebot-helmsman/round-robin-scheduler-tested ... [MRG+1] Downloader-aware Priority Queue for Scrapy	2019-04-07 00:06:08 +05:00
Maram Sumanth	aa46e1995c	[MRG+1] Show elapsed time in statscollector (#3638) ... * Update corestats.py * Update corestats.py * corrected tests * Update corestats.py * Update scrapy/extensions/corestats.py	2019-04-07 00:03:40 +05:00
Kostis Anagnostopoulos	3a493b6066	fix: do not catch system exceptions like KeyboardInterrupt	2019-04-05 11:52:00 +02:00
Konstantin Lopuhin	b5c552d17f	Merge pull request #2061 from elacuesta/process_spider_exception_generator ... [MRG+1] process_spider_exception on generators	2019-04-01 10:43:19 +03:00
Eugenio Lacuesta	f5e0b6b89a	parse command: rename cb_kwargs option to cbkwargs	2019-03-29 14:03:26 -03:00
Eugenio Lacuesta	e8af6331b5	Add cb_kwargs option to the parse command	2019-03-28 15:03:40 -03:00
Eugenio Lacuesta	70a4d93aa3	Callback kwargs: more tests	2019-03-28 13:28:19 -03:00
Mikhail Korobov	ec719f55e7	Merge pull request #3682 from elacuesta/rule_process_request_response_parameter ... [MRG+1] Rule.process_request: access Response object	2019-03-28 01:39:54 +05:00
Eugenio Lacuesta	c43a23152b	Merge remote-tracking branch 'upstream/master' into callback_kwargs	2019-03-27 14:42:41 -03:00
Maram Sumanth	2fd8b7c28c	[MRG+1] redirect_reasons in Request.meta (#3687)	2019-03-27 00:15:53 +05:00
Adrián Chaves	ba13de29e1	Merge branch 'master' into feat-685	2019-03-26 10:38:38 +01:00
Vostretsov Nikita	73e4ff5304	report warnings	2019-03-25 13:48:58 +00:00
Vostretsov Nikita	103fdc40c1	Merge remote-tracking branch 'upstream/master' into round-robin-scheduler-tested	2019-03-25 16:16:42 +03:00
Vostretsov Nikita	31b8a6b33a	report warnings	2019-03-25 16:08:40 +03:00
Lucy Wang	df574de8cc	improve tests and fix some lint warnings (#6) ... * refactor downloader-aware test cases * fix lint * add doctest for _path_safe * remove unused code * better doctest	2019-03-25 16:08:40 +03:00
Vostretsov Nikita	8afffb7234	Tests Cleanup ... add doctest for function no need in this variables move common assertion inside function rename variable rename variables rename function use function this is not a method of public API correct name for test Update docs/topics/settings.rst Co-Authored-By: whalebot-helmsman <whalebot.helmsman@gmail.com>	2019-03-25 16:08:40 +03:00
Vostretsov Nikita	989bba6cb3	Revert "new signal" ... This reverts commit 646164fd7d6dd52061804d2df7424cff929bf739. remove tests Revert "emit new signal" This reverts commit fcde0c6880678957a76af6083b6248f430a00fcf. Revert "documentation for new signal" This reverts commit 8aeb9f696ece95c16499a96767a7afa3d9c4abf4.	2019-03-25 16:08:40 +03:00
Vostretsov Nikita	443fb98a47	Use downloader directly ... rename variable remove old write function remove unused imports remove old read function remove unused function use mock methods mock downloader close downloader add parse method use new PQ class create mock downloader use downloader directly remove mark/unmark mechanism	2019-03-25 16:08:40 +03:00
Vostretsov Nikita	3b1db71dac	New signal ... update signature documentation for new signal utilize new signal correct signal handler signature emit new signal test another signal new signal rename test file faster test rename test case tests for signal emitting in bad cases	2019-03-25 16:08:40 +03:00
Vostretsov Nikita	757f53a324	Address Lucy's comments ... add tests to check correctness of slot setermination unmark requests after downloading shorter better exception message	2019-03-25 16:08:40 +03:00
Mikhail Korobov	90934959d0	actually apply __slots__ suggestion ... [wip] refactoring * SlotPriorityQueues doesn't care about objects inside, it is now just a container for multiple priority queues * assorted variable renames * don't inherit DownloaderAwarePriorityQueue from SlotBasedPriorityQueue * apply @whalebot-helmsman's suggestions for __slots__ and meta issues more bike-shedding * remove mutable default arguments * more verbose variable names remove unneeded code * PriorityAsTupleQueue.is_empty does the same as len(self) == 0 * custom PriorityAsTupleQueue.close is not needed after a switch to namedtuples * is_new and is_empty return values are unused * "url" local variable is unused PrioritySlot.__str__ shouldn't return unicode in Python 2 also, do some bike-shedding: _pathable -> _path_safe use namedtuple for PrioritySlot cleanup: _get_from_request does the same here Request.meta is always a dict	2019-03-25 16:08:40 +03:00
Eugenio Lacuesta	1b4385b7e3	Rule.process_request: move deprecation warnings and compiling code, update tests	2019-03-22 19:46:17 -03:00
Mikhail Korobov	729580dae2	Merge pull request #3671 from Gallaecio/item-deepcopy ... Implement Item.deepcopy()	2019-03-22 22:56:39 +05:00
Adrián Chaves	7c148fce5a	Implement Item.deepcopy()	2019-03-22 17:07:10 +01:00
Konstantin Lopuhin	af2b666595	Merge pull request #3505 from kasun/json_request ... [MRG+2] Request subclass for json requests #3504	2019-03-22 18:25:54 +03:00
Vostretsov Nikita	987c2ae4a9	test ip concurrency incompatibility with DAPQ	2019-03-22 15:40:37 +03:00
Vostretsov Nikita	4163a7a1c7	no need for this	2019-03-22 15:40:37 +03:00
Vostretsov Nikita	d970be64cc	Integration test ... integration testing only everything is working, not logic of PQ use method create slot attribute in constructor corect class for test case stop crawler in teardown method use class correct entity naming python 2 adaptation integration test with crawler and spider	2019-03-22 15:40:36 +03:00
Vostretsov Nikita	a23e1894b3	Fix boto problem ... another way to fix boto problem Revert "fix for travis ci based on https://github.com/boto/boto/issues/3717" This reverts commit 150d2564ff0ea994652da7f5be333d72e0b38d93. fix for travis ci based on https://github.com/boto/boto/issues/3717	2019-03-22 15:40:36 +03:00
Vostretsov Nikita	f56079f6c7	Test cleanups ... PEP8 fixes no need to close implicitly do not use pytest need to put it into class remove round-robin queue additional check for empty queue use pytest tmpdir fixture	2019-03-22 15:40:36 +03:00
Vostretsov Nikita	821f5bb260	First implementation ... handle exception use O(N) instead of O(NlogN) here we have request as struct additional check for meptiness small performance improvement do not consume another request test number of responses mark requests back to 3 slots test case raise exceptions in case of missed meta add marks to requests and work only with your own requests only disk queue should obtain signals separate functions for slot rasd/write use signlas without variable stop crawler get signals in correct place logic test for download-aware priority queue update comment for structure ensure text type transform slot name to path use implicit structure use unicode type implicitly use real crawler add signals more slot accounting simple implementation of pop small slot accounting code no need for custom len function ability to call super in py27 add slots generic tests for downloader aware queue dummy implementation of crawler aware priority queue move common logic to base class rename class pass crawler to pqclass constructor do not copy quelib.PriorityQueue code add comment about new class remove obsolete function modify behaviour of queuelib.PriorityQueue to dodge very complex priority better way to get name remove obsolete commentary check boundaries function for priority convertion with known limits correct import path move file do not switch on by deffault as ip concurrency not supported set scheduler slot in case of empty slot use constant single place for added urls single place for constants use as default queue correct format for error text test migration from old version with on disk queue in these tests we have only two inflection points - jobdir and priority_queue_cls we do not need separate mock spider, use usual one do not rely on order of dict elements, imply order of list test round robiness of priority queue add comments and requirements for our magick function remove debug logging put queues into slot as we fabricate priorities we do not need special types anymore fabricate priority for priority queue more versatile priorities Scheduler class is not inflection point wrap correct types check for emptinees before initialization tests for new priority queue correct default type for startprios use exact values put common settings to base class test priorities for disk scheduler test dequeue for disk scheduler test length for disk scheduler setUp/tearDown methods for on disk schedulers new methods remove excessive line base class to handle scheduler creation correct method names test priorities deque test close scheduler on test end enqueue some requests test template for scheduler use downloader slot I/O implementation for RoundRobin queue round-robin implementation without I/O and slot detection wrappers for every disk queue class	2019-03-22 15:40:36 +03:00
Eugenio Lacuesta	6760bca74b	Rename Request.kwargs to Request.cb_kwargs	2019-03-15 22:42:29 +00:00
Eugenio Lacuesta	645e8d16a4	Count keyword argument checks	2019-03-15 22:20:36 +00:00
Eugenio Lacuesta	01ed605d02	PEP8 changes to test_spider.py	2019-03-15 16:54:14 +00:00
Victor Torres	fda1d04b81	Merge branch 'master' into feed-storage-s3-acl	2019-03-15 13:26:45 -03:00
Adrián Chaves	70a0f1f2e4	Merge pull request #3610 from Gallaecio/spidercls-check ... [MRG+1] Check that spidercls arguments in scrapy.crawler classes are not spid…	2019-03-15 08:25:16 +01:00
Adrián Chaves	1f7413dc88	Merge pull request #3635 from matthieucham/feature-filteringlinkextractor-restrict-text ... [MRG+1] Feature filteringlinkextractor restrict text	2019-03-15 07:58:28 +01:00
Mikhail Korobov	5dc94db847	Merge branch 'master' into backward	2019-03-14 22:21:09 +05:00
Eugenio Lacuesta	22fda61d62	Rule.process_request: tests	2019-03-13 10:54:38 +00:00
Adrián Chaves	75d6f56c8a	Switch from ` to `` where inline code formatting is desired	2019-03-01 16:56:58 +01:00
Adrián Chaves	858f5be747	backwards → backward (adj.)	2019-03-01 16:10:23 +01:00
Matthieu Grandrie	e3b15252c8	New constructor arg *restrict_text* for FilteringLinkExtractor. ... Same as allow and deny args, it holds a string, a regex or an iterable of. Links whose text don't match one of the regex are filtered out. DOC restrict_text in LxmlLinkExtractor	2019-02-28 17:21:17 +01:00
Júlio César Batista	0bb3d8ca93	Updating Google Cloud Storage scheme to gs instead of gcs	2019-02-27 18:41:01 -03:00
Victor Torres	9b8ba4c383	try to import botocore before runing some tests	2019-02-14 16:20:56 -02:00
Victor Torres	ea8be627d1	botocore is not supported on debian jessie	2019-02-13 20:32:36 -02:00
Victor Torres	dc0b643832	refactoring tests to avoid mocking private method	2019-02-13 19:44:50 -02:00
Adrián Chaves	03e61b9908	Check that spidercls arguments in scrapy.crawler classes are not spider objects	2019-02-12 14:46:27 +01:00
Victor Torres	1eac2a163c	simplifying how we deal with threads.deferToThread calls	2019-02-08 16:50:39 -02:00