scrapy

Commit Graph

Author	SHA1	Message	Date
Andrey Rakhmatullin	61f99f2df1	More granular untyped defs checking in tests. (#7712 )	2026-07-08 10:05:45 +02:00
Andrey Rakhmatullin	6fe27ba33e	Add more no-reactor tests (#7259 ) * Generic changes and scrapy bench. * scrapy check. * scrapy crawl. * scrapy fetch. * scrapy parse. * scrapy runspider. * scrapy shell. * Skip httpx tests on default-reactor. * Review requires_reactor marks. * Make test functions that require an event loop async def. * Improve test_pending_asyncio_tasks(). * Add Mac OS DNS error. * Refactor most of test_scheduler.py. * Finish refactoring DownloaderAwareSchedulerTestMixin. * Refactor test_engine_loop.py. * Add the no-reactor-extra-deps tox env, run no-reactor on macos. * Skip unhandled CancelledError when shutting down the engine. * Fix typing and pre-commit checks. * Fix typing problems in master. --------- Co-authored-by: Adrian <adrian@zyte.com>	2026-03-24 15:29:22 +05:00
Andrey Rakhmatullin	9f4651151d	Deprecate ScrapyCommand.set_crawler(), remove/fix some dead code. (#7276 )	2026-03-24 10:36:48 +01:00
Andrey Rakhmatullin	939db88b04	Disable mypy allow_untyped_calls. (#7331 )	2026-03-24 10:31:17 +01:00
Andrey Rakhmatullin	7010985e4f	Bump ruff, fix some of rules (#7277 )	2026-02-23 11:48:38 +01:00
Andrey Rakhmatullin	3186ccf5d5	Reactorless tests 1: the test env (#6952 ) * Reactorless tests. * Updates after merging master. * Update a new import. * Updates after merging master. * Update testenv:no-reactor-pinned. * Foundations for the reactorless mode. * Fixes after merging master. * Updates after merging reactorless-base. * Add simple subprocess tests for reactorless AsyncCrawler. More reactorless tests. * Refactor AsyncCrawlerProcess.start(). * More checks. * Fix test_reactorless_import_hook. * More tests. * Call install_reactor() before asyncio.run(). * Cleanup. * Rephrase. * Fix a test. * Fixes. * Rephrasing. * Set TELNETCONSOLE_ENABLED=False in the reactorless mode. * Fix a new import. * Clarify --reactor marks. * Cleanup. * More docs/comments. * Update test_fallback_workflow(). * Update TestDeferredFFromCoroF. * Typing improvements.	2026-02-02 19:25:30 +05:00
Andrey Rakhmatullin	d414d393d4	Remove Python 3.9 support (#7121 )	2025-10-27 12:37:49 +01:00
Andrey Rakhmatullin	d825133284	Reduce deps on unittest, unify inlineCallbacks imports in tests. (#6873 )	2025-06-06 22:59:09 +02:00
Andrey Rakhmatullin	d0dabbc097	Enable AsyncioSelectorReactor by default. (#6713 ) * Enable AsyncioSelectorReactor by default. * Improve get_crawler(), switch more tests to it. * Fix the remaining default-reactor test failures. * Address documentation feedback. * Make pinned envs more consistent.	2025-03-12 01:18:30 +05:00
Andrey Rakhmatullin	a898331d14	Preparations for switching to direct pytest. (#6678 )	2025-02-16 20:28:58 +01:00
Andrey Rakhmatullin	7dd92e6e43	Add pylint rules to ruff, refresh the ignore list of pylint itself (#6584 )	2024-12-16 09:44:36 +01:00
Rotzbua	c7b2b097b1	fix(typo): correct `successfully`	2024-01-05 10:50:45 +01:00
Andrey Rakhmatullin	3756216339	Rename methods.	2023-09-13 15:47:22 +04:00
Andrey Rakhmatullin	bb15c93a2b	Add Crawler._load_settings().	2023-09-12 14:44:16 +04:00
Andrey Rakhmatullin	036d5836d0	Move more things from Crawler.__init__() to .crawl().	2023-09-08 19:29:05 +04:00
Georgiy Zatserklianyi	e9b088f1fb	periodic_log: typing	2023-08-04 12:49:22 +03:00
Georgiy Zatserklianyi	6fd94fdcb3	periodic_log: tests updated (errors fixed)	2023-06-28 19:55:33 +03:00
Georgiy Zatserklianyi	56c38231b4	periodic_log: tests for logging stats added	2023-06-27 11:48:51 +03:00
Georgiy Zatserklianyi	ebce5b4bcb	periodic_log: tests for logging deltas added	2023-06-27 09:22:50 +03:00
Georgiy Zatserklianyi	2ce4856508	periodic_log: Exception handling on init updated	2023-06-25 12:16:59 +03:00
Georgiy Zatserklianyi	6e65eeb07b	periodic_log: tests [wip] added	2023-06-16 14:30:50 +03:00

21 Commits