Commit Graph

234 Commits

Author SHA1 Message Date
Adrian c285f4cb18
Support CloseSpider during spider startup (#7905) 2026-08-10 10:06:54 +02:00
Adrian 508bd7faec
Type late Crawler attributes as always set instead of None (#7882) 2026-08-10 09:07:34 +02:00
Adrian bee31890a2
Report exceptions from Spider.start() (#7884) 2026-08-09 22:56:03 +05:00
Adrian 25e6884e2f
Address recursion and media ignore-request reporting issues (#7673)
* Address recursion and media ignore-request reporting issues

* Improve test coverage

* Address feedback
2026-07-27 20:34:10 +05:00
Albert Eduardovich N. 9776a72a6a
chore: simplify some code, get rid of nested fns where it's makes sence + pylint (#7401)
* chore: simplify some code, get rid of nested fns where it's makes sence

reasoning: https://wemake-python-styleguide.readthedocs.io/en/latest/pages/usage/violations/best_practices.html#wemake_python_styleguide.violations.best_practices.NestedFunctionViolation

* chore: address some pylint rules

* remove `pylint: disable=deprecated-class` from `reactor.py`

* fix typo

* fix another typo :)

* revert backward incompatible arg renames

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2026-05-04 23:32:46 +05:00
Andrey Rakhmatullin 6fe27ba33e
Add more no-reactor tests (#7259)
* Generic changes and scrapy bench.

* scrapy check.

* scrapy crawl.

* scrapy fetch.

* scrapy parse.

* scrapy runspider.

* scrapy shell.

* Skip httpx tests on default-reactor.

* Review requires_reactor marks.

* Make test functions that require an event loop async def.

* Improve test_pending_asyncio_tasks().

* Add Mac OS DNS error.

* Refactor most of test_scheduler.py.

* Finish refactoring DownloaderAwareSchedulerTestMixin.

* Refactor test_engine_loop.py.

* Add the no-reactor-extra-deps tox env, run no-reactor on macos.

* Skip unhandled CancelledError when shutting down the engine.

* Fix typing and pre-commit checks.

* Fix typing problems in master.

---------

Co-authored-by: Adrian <adrian@zyte.com>
2026-03-24 15:29:22 +05:00
Andrey Rakhmatullin e02ad08672
Move to mypy --strict with exceptions. (#7300)
* Move to mypy --strict with exceptions.

* Fix typing-tests.
2026-03-02 15:47:23 +05:00
Andrey Rakhmatullin 7010985e4f
Bump ruff, fix some of rules (#7277) 2026-02-23 11:48:38 +01:00
Andrey Rakhmatullin 16929c0991
Exclude deprecated code from coverage. (#7241) 2026-02-06 09:15:07 +01:00
Andrey Rakhmatullin 0f07b2e38c
Pass the spider arg to custom stat collectors {open,close}_spider(). (#7213) 2026-01-09 16:57:53 +01:00
Andrey Rakhmatullin 7bef98b4f1
Release notes for 2.14.0 (#7035)
* Initial release notes for 2.14.0, up to ed63fa9.

* Cover 2.14 in the release notes up to 393d715.

* Cover 2.14 in the release notes up to eb49647.

* Cover 2.14 in the release notes up to 426aafd.

* Cover 2.14 in the release notes up to 1e8de24.

* Cover 2.14 in the release notes up to 5a7e132.

* Bump sphinx-lint.

* Finalize the 2.14.0 release notes.

* Drop more of the old versionadded directives.

* Address feedback.
2026-01-05 14:51:05 +05:00
Andrey Rakhmatullin 9bfa58e36c
Deprecate send_catch_log_deferred(). (#7161)
* Add a test for not having pending tasks.

* Refactor TestFeedExporterSignals.

* Refactor FeedExporter.close_spider().

* More engine start/stop robustness.

* Refactor send_catch_log_async(), deprecate send_catch_log_deferred().

* Add pragma: no cover.

* Warn on signal handlers returning a Deferred.

* Make _pending_close_coros an instance attribute.

* Remove an unused function.

* Remove the outdated TODO.
2025-12-10 14:42:49 +05:00
Andrey Rakhmatullin b68330811b
Deprecate spider arguments of StatsCollector methods (#7011) 2025-08-20 09:52:22 +02:00
Andrey Rakhmatullin d3e15a10cf
Add ItemPipelineManager.process_item_async() and ensure_awaitable(). (#7005)
* Add ItemPipelineManager.process_item_async() and ensure_awaitable().

* Wording.
2025-08-12 19:57:37 +05:00
Andrey Rakhmatullin 57f539d7a5
Deprecate Crawler.stop(). (#6999) 2025-08-11 08:32:55 +02:00
Andrey Rakhmatullin a0b766f9e1
More *_async APIs (#6997)
* Add ExecutionEngine.close_spider_async().

* Add Scraper.{open,close}_spider_async().

* Fix double engine stopping via Crawler.stop().

* Add ExecutionEngine.stop_async().

* Add versionadded to new async APIs.

* Add ExecutionEngine.close_async().
2025-08-08 22:15:41 +05:00
Andrey Rakhmatullin a3daa3612e
Rewrite ExecutionEngine.close_spider() to inlineCallbacks. (#6986) 2025-08-06 08:01:55 +02:00
Andrey Rakhmatullin baa579df62
Deprecate various `spider` args in `MiddlewareManager` subclasses (#6984)
* Add a crawler argument to MiddlewareManager.__init__().

* Deprecate the spider argument in some MiddlewareManager subclasses.

* Remove the spider argument from ExecutionEngine.open_spider_async().

* Remove the spider argument from Scraper.open_spider().

* Fix the Scraper.close_spider() type hint.

* Remove the spider argument from Downloader internals.

* Deprecate the spider arg of download_func.

* Deprecate the spider arg of Downloader.fetch().

* Deprecate the spider arg of ItemPipelineManager.process_item().

* Fix pylint.

* Tests for MiddlewareManager spider code.

* Deprecated spider arg handling for custom pipeline managers.

* Add a test for the deprecated spider arg in a custom downloader.

* Remove TODO.

* Fix pylint.
2025-08-05 12:21:08 +05:00
Andrey Rakhmatullin 32f2aede82
Deprecate non-async counterparts of new async APIs. (#6979)
* Deprecate non-async counterparts of new async APIs.

* Deprecate Scraper.call_spider().

* Add _schedule_coro().
2025-08-01 00:58:48 +05:00
Felipe Benevolo 89f53f0555
Added download_async() to ExecutionEngine and test cases for download() and download_async() (#6842)
* added async_download()

* add case tests for ExecutionEngine download() and async_download()

* change async_download() to download_async() in Engine class and subsequent tests

* fixed reactor import in test_engine.py

* Simplify TestEngineDownloadAsync.

* replaced tearDown by self.engine.downloader.close in test_engine.py and replace @defer.inlineCallbacks by @inlineCallbacks

* Simplify ExecutionEngine.download().

* Remove an unneeded import.

* Refactor ExecutionEngine.download{,_async}() tests.

* Deprecate ExecutionEngine.download().

* Update doc examples to use download_async().

* Replace uses of download() with download_async().

* Fix test_downloadermiddleware_robotstxt.py.

* Remove a stray file.

---------

Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-07-27 23:17:51 +05:00
Andrey Rakhmatullin df342eee6e
Bump tool versions. (#6941) 2025-07-05 10:30:30 +02:00
Andrey Rakhmatullin 7842180b2f Merge remote-tracking branch 'origin/2.13' 2025-07-01 17:28:12 +05:00
Andrey Rakhmatullin c9cdf0af3c
Narrow down TestEngine::test_short_timeout() expectations. (#6911) 2025-07-01 13:47:43 +02:00
Andrey Rakhmatullin ef01a953b1 Fix setting ExecutionEngine._slot to None on close. (#6910)
* Fix setting ExecutionEngine._slot to None on close.

* Check self._slot in needs_backout().
2025-06-25 17:30:57 +05:00
Andrey Rakhmatullin ff7795b159
Fix setting ExecutionEngine._slot to None on close. (#6910)
* Fix setting ExecutionEngine._slot to None on close.

* Check self._slot in needs_backout().
2025-06-25 17:29:29 +05:00
Andrey Rakhmatullin 7fbd56bc9b Handle exceptions in _start_request_processing(), cancel it on engine stop (#6900) 2025-06-23 22:38:38 +05:00
Andrey Rakhmatullin 0d75355b41
Handle exceptions in _start_request_processing(), cancel it on engine stop (#6900) 2025-06-23 19:36:54 +02:00
Andrey Rakhmatullin d329eedfef Don't try to close ExecutionEngine.downloader when it doesn't exist. (#6867) 2025-06-06 16:26:38 +05:00
Andrey Rakhmatullin 657e6cb2b5
Don't try to close ExecutionEngine.downloader when it doesn't exist. (#6867) 2025-06-06 13:02:15 +02:00
Andrey Rakhmatullin 8fb8d2c6b8
Add AsyncioLoopingCall. (#6855) 2025-06-03 20:21:33 +02:00
Andrey Rakhmatullin 05b3b205ce
Add `AsyncCrawlerProcess` and `Crawler.crawl_async()` (#6817)
* Add a basic Crawler.crawl_async().

* Add custom loop tests for *CrawlerRunner.

* Add AsyncCrawlerProcess.

* Update related docs.

* Update practices.rst.

* Address test failures.

* Add a note about AsyncCrawler* to the docs about switching reactors.

* Address feedback.

* Update for TID253.

* Simplify test_crawler_crawl_async_twice_parallel_unsupported().
2025-05-28 18:55:44 +05:00
Andrey Rakhmatullin 1ddcb568e2 Add send_catch_log_async(). 2025-05-15 15:45:55 +05:00
Andrey Rakhmatullin b86f00327a
Refactor more Deferred functions. (#6795) 2025-05-13 20:47:57 +02:00
Adrián Chaves 036f3e5627
Support asynchronous start requests (#6729) 2025-05-07 19:04:03 +02:00
Andrey Rakhmatullin eced5ca2d3 Remove the unnecessary spider argument from Scraper and related code. 2025-04-30 00:57:44 +05:00
Adrián Chaves 5a0690c89d
Remove or post-pone the use of itemadapter.is_item, as a potentially expensive call (#6719) 2025-03-11 14:52:48 +01:00
Andrey Rakhmatullin c87354cd46 Enable SIM Ruff rules. 2025-01-02 01:03:36 +05:00
Andrey Rakhmatullin d47f142d0f Add flake8-type-checking rules to ruff. 2024-12-10 01:28:20 +05:00
Andrey Rakhmatullin b042ad255d
Address some previously ignored pylint messages. (#6531) 2024-11-11 11:49:52 +01:00
Andrey Rakhmatullin c9095ef927 Remove --keep-runtime-typing from pyupgrade. 2024-10-17 21:26:02 +05:00
Andrey Rakhmatullin f65e64a724
Misc typing improvements. (#6494) 2024-10-16 18:38:43 +02:00
Vsevolod Breus 5391663072
Drop Python 3.8 Support (#6472) 2024-10-16 10:03:16 +02:00
LucasSD f260f819e0 Remove debug log message from _schedule_request method 2024-09-09 20:26:02 +01:00
Georgiy Zatserklianyi 6ce0342beb
Allow yielding items from start_requests (#6417)
Co-authored-by: Georgiy Zatserklianyi <george.zatseklyany@gmail.com>
Co-authored-by: Adrián Chaves <adrian@chaves.io>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2024-08-26 20:53:06 +02:00
Andrey Rakhmatullin d08f559600
Add flake8-type-checking. (#6413) 2024-06-25 10:20:59 +02:00
Andrey Rakhmatullin 1282ddf8f7
Add parameters to most Deferred in scrapy/core. (#6395) 2024-06-10 10:27:50 +02:00
Andrey Rakhmatullin 2e214210f6 Add parameters to iterable generics, replace generators with iterables. 2024-06-05 21:44:04 +05:00
Andrey Rakhmatullin 019f23e3b7 Add parameters to some of typing.Callable. 2024-06-02 18:42:01 +05:00
Andrey Rakhmatullin 751c91e614 Add parameters to misc generics. 2024-05-31 21:57:14 +05:00
Adrián Chaves d2f1e00a6a
Merge 2.11.2 changes (#6363) 2024-05-14 18:54:11 +02:00