scrapy

Commit Graph

Author	SHA1	Message	Date
Andrey Rakhmatullin	b2b2d0b015	Rename TWISTED_ENABLED to TWISTED_REACTOR_ENABLED. (#7394 )	2026-04-02 10:20:54 +02:00
Andrey Rakhmatullin	6fe27ba33e	Add more no-reactor tests (#7259 ) * Generic changes and scrapy bench. * scrapy check. * scrapy crawl. * scrapy fetch. * scrapy parse. * scrapy runspider. * scrapy shell. * Skip httpx tests on default-reactor. * Review requires_reactor marks. * Make test functions that require an event loop async def. * Improve test_pending_asyncio_tasks(). * Add Mac OS DNS error. * Refactor most of test_scheduler.py. * Finish refactoring DownloaderAwareSchedulerTestMixin. * Refactor test_engine_loop.py. * Add the no-reactor-extra-deps tox env, run no-reactor on macos. * Skip unhandled CancelledError when shutting down the engine. * Fix typing and pre-commit checks. * Fix typing problems in master. --------- Co-authored-by: Adrian <adrian@zyte.com>	2026-03-24 15:29:22 +05:00
Andrey Rakhmatullin	e48a1bdde3	Speed up and refactor command tests (#7118 )	2025-10-27 12:28:16 +01:00
Andrey Rakhmatullin	ed63fa94d6	Deprecate spider args of middlewares and pipelines. (#7006 ) * Deprecate spider args of spider middleware methods. * Deprecate the spider arg of pipeline process_item(). * Fix a typing issue. * Deprecate the spider arg of pipeline {open,close}_spider(). * Cleanup. * Update docs. * Add pragma: no cover to some deprecated code. * More tests. * More tests. * More tests. * Simplify _process_parallel().	2025-08-21 13:23:10 +05:00
Andrey Rakhmatullin	8c8f4ff033	Extract in-process test HTTP servers into separate processes (#6960 ) * Switch TestEngineBase to mockserver. * Split mockserver.py. * Stop using WrappingFactory. * Remove keyfile/certfile fields from TestHttpBase. * Move webclient-only resources to test_webclient.py. * Refactor MockServer.url(). * Migrate TestHttpBase to mockserver. * Migrate TestSimpleHttpsBase to a separate mockserver. * Rename TestHttpMockServerBase. * Migrate TestHttpProxyBase to a separate mockserver. * Refactor HTTP mockservers.	2025-07-25 12:42:27 +05:00
Andrey Rakhmatullin	92c18d15b4	Remove ProcessTest and SiteTest. (#6885 ) * Remove ProcessTest and SiteTest. * Restore the support for Windows line endings in TestParseCommand. * Add a test for running a scrapy command in a project subdir. * Remove pywin32 from test deps.	2025-06-13 22:18:27 +05:00
Andrey Rakhmatullin	d825133284	Reduce deps on unittest, unify inlineCallbacks imports in tests. (#6873 )	2025-06-06 22:59:09 +02:00
Andrey Rakhmatullin	5a605969bd	Converting tests to plain asserts, part 2. (#6699 )	2025-03-06 20:52:41 +01:00
Andrey Rakhmatullin	a5731c1944	Move most of the test utils inside tests.	2025-03-02 21:04:12 +05:00
Adrián Chaves	e8dadb9592	scrapy parse: fix the signature of callbacks from the CLI (#6182 )	2024-01-15 13:37:03 +01:00
Mohammadtaher Abbasi	afafc2781a	Fix parse command issues with asyncio (#5824 )	2023-03-10 16:20:15 +01:00
Jalil SA	a5c1ef8276	sort imports with isort	2023-01-25 14:25:15 -06:00
Emmanuel Rondan	e211ec0aa2	adding black formatter to all the code	2022-11-29 11:30:46 -03:00
Mark Mayo	f9a29f03d9	Address some issues reported by Pylint (#5677 ) Co-authored-by: Felipe Boff Nunes <51033921+felipeboffnunes@users.noreply.github.com> Co-authored-by: Andrey Rahmatullin <wrar@wrar.name>	2022-11-27 11:00:13 +01:00
Andrey Rahmatullin	d19a216e10	Merge branch 'master' into pathlib	2022-11-25 18:20:25 +05:00
Matthew Donoughe	40d9ca3bdd	use pathlib	2022-10-17 17:40:10 -04:00
Mohammadtaher Abbasi	043575123c	Add async callback support to the parse command (#5577 )	2022-10-15 10:11:05 +02:00
Andreas Tziortziortziopoulos	b5c15d87ff	[issue3264] Separate test for not matched spider to a url	2022-05-22 12:19:20 +03:00
Andreas Tziortziortziopoulos	83c1939281	Issue #3264 , fix error handling when spider is not matched Changes Implementation: - Check whether Spider exists or is None, and if it's None skip execution of start_requests() with non existing Spider Testing: - Add a test case with invalid url inside test_command_parse Test proves that non-matched Spider does not throw an AttributeError	2022-05-06 03:59:30 +03:00
Alex	e2e2ffd0d1	Move from optparse to argparse (#5374 )	2022-02-09 20:52:07 +01:00
Ammar Najjar	58ca8bbf6d	Use f-strings (#4307 )	2020-08-26 07:11:05 +02:00
Marc Hernández	336f19f5cc	Change super syntax (#4707 )	2020-08-04 20:42:01 +02:00
Aditya Kumar	3f7e8635f4	Allow the parse command to write data to a file (#4377 )	2020-07-11 08:48:24 +02:00
Eugenio Lacuesta	9c6d13075a	Flake8: Remove E128 (tests, part 1)	2020-05-06 18:56:14 -03:00
elacuesta	dfbe1d9507	Remove object base class (#4430 )	2020-03-16 20:12:46 +01:00
Marc Hernandez Cabot	6fb85951ce	fix E22X flake8	2020-02-21 08:39:14 +01:00
Marc Hernández	91bbc70bc1	fix E30X flake8 (#4355 )	2020-02-21 06:05:31 +01:00
Adrián Chaves	f261cf65e9	Add missing blank lines between functions and classes Also fixed 2 unrelated Flake8 issues	2019-11-18 17:16:09 +01:00
Andrey Rakhmatullin	a138fb05d4	Replace to_native_str calls with to_unicode.	2019-11-14 13:45:39 +05:00
s-sanjay	2c14692e60	remove .keys() to avoid creating a tmp list/keyview obj (#4031 ) Also add --verbose and --nolinks for code coverage	2019-09-27 09:56:43 +02:00
Eugenio Lacuesta	b714a372e2	[Tests] Prevent more DeprecationWarnings * assertRegexpMatches -> assertRegex * invalid escape sequence \[	2019-07-13 20:23:15 -03:00
Eugenio Lacuesta	f5e0b6b89a	parse command: rename cb_kwargs option to cbkwargs	2019-03-29 14:03:26 -03:00
Eugenio Lacuesta	e8af6331b5	Add cb_kwargs option to the parse command	2019-03-28 15:03:40 -03:00
Daniel Grana	96517cb7de	Fix test_command_parse under windows	2018-08-15 01:08:40 -07:00
IAlwaysBeCoding	846fd83512	removed commented out code, wrapped line to pep-8 and removed backlashes	2017-11-11 18:30:01 -05:00
Joakim Uddholm	8c38dde4e8	Moved parse command tests to its own file. Added some checks to check for logged errors.	2016-09-19 05:33:05 +02:00

36 Commits