71df815fcd
Clarify spider middleware handling of callback requests
...
Make explicit that process_spider_output() handles both items and
requests from callbacks before they reach pipelines or scheduler.
2025-12-31 13:23:50 +01:00
4e175e7930
Add spider middleware start request processing to lifecycle docs
...
Address reviewer feedback that spider middleware coverage was incomplete.
Changes include:
- Add new "Start request processing" section explaining process_start()
- Update lifecycle overview to include start processing as phase 2
- Update spider middleware section to show both integration points
- Update sequence diagram to show START PHASE and CRAWL PHASE
- Add missing signal mentions (engine_started, spider_opened,
request_dropped, spider_closed)
2025-12-31 13:20:45 +01:00
5941413af2
Add lifecycle documentation to toctree
...
Fixes sphinx warning about orphaned document by adding lifecycle.rst
to the Extending Scrapy section of the documentation index.
2025-12-30 08:35:22 +01:00
acabf82bde
Add Request to Response lifecycle documentation
...
Document the end-to-end flow of requests through Scrapy's internals,
including the engine's orchestration role, scheduling, downloading,
spider callback execution, and item pipeline processing.
This conceptual guide complements the existing architecture overview
by providing a detailed narrative of the request lifecycle.
2025-12-29 15:16:42 +01:00
9381ad893d
Remove outdated version marker directives (< 2.8.0) from documentation ( #7198 )
2025-12-22 10:00:39 +01:00
180ca39b23
Deprecate returning Deferreds from pipeline methods ( #7179 )
...
* Add tests for exceptions in pipelines.
* Deprecate returning Deferreds from pipeline process_item().
* Deprecate returning Deferreds from pipeline {open,close}_spider().
* Update the custom pipeline docs.
2025-12-16 13:35:07 +05:00
c49ae2115a
Refactor MediaPipeline.process_item() to async def ( #7177 )
2025-12-15 10:02:04 +01:00
588f3d4f65
Remove docs mentioning deprecated spider attrs. ( #7176 )
2025-12-15 09:51:56 +01:00
d8583a89c7
Change default priority queue to DownloaderAwarePriorityQueue ( #6940 )
...
* Change default priority queue to DownloaderAwarePriorityQueue
* Fix documentation building
* Simplify test_start_already_running_exception changes.
* Modernize the test.
* Fix TestEngineCloseSpider.
* Fix typing.
* Remove special slot=None handling.
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-12-11 11:25:17 +01:00
9bfa58e36c
Deprecate send_catch_log_deferred(). ( #7161 )
...
* Add a test for not having pending tasks.
* Refactor TestFeedExporterSignals.
* Refactor FeedExporter.close_spider().
* More engine start/stop robustness.
* Refactor send_catch_log_async(), deprecate send_catch_log_deferred().
* Add pragma: no cover.
* Warn on signal handlers returning a Deferred.
* Make _pending_close_coros an instance attribute.
* Remove an unused function.
* Remove the outdated TODO.
2025-12-10 14:42:49 +05:00
5105f55a98
Add `DownloaderMiddlewareManager.download_async()`, deprecate returning Deferreds from downloader mws ( #7069 )
...
* Restore coverage in MiddlewareManager spider arg handling.
* Move MiddlewareManager._warn_spider_arg() tests.
* Add DownloaderMiddlewareManager.download_async().
* Make download_func async.
* Inline MiddlewareManager._warn_spider_arg().
* Deprecate returning deferreds from downloader middlewares.
* Update the downloader mw doc about async def.
* Use global_object_name() instead of repr().
* Update docs/topics/downloader-middleware.rst
Co-authored-by: Adrián Chaves <adrian@chaves.io>
* Update downloader-middleware.rst
---------
Co-authored-by: Adrián Chaves <adrian@chaves.io>
2025-12-09 20:07:36 +05:00
0adc561348
Add a list of remaining Deferred APIs. ( #7146 )
2025-11-27 20:13:04 +05:00
4cb0144b39
Merge tag '2.13.4'
...
Bump version: 2.13.3 → 2.13.4
2025-11-17 16:47:54 +05:00
2f62ab532d
Bump version: 2.13.3 → 2.13.4
2025-11-17 16:30:37 +05:00
31a9c03c24
Release notes for 2.13.4. ( #7144 )
...
* Release notes for 2.13.4.
* Formatting.
2025-11-17 16:30:04 +05:00
d091256c58
Remove the deprecated sphinx-hoverxref ( #6922 )
2025-11-14 15:39:21 +05:00
804ae167df
Update tool versions ( #7127 )
2025-10-27 14:11:31 +01:00
61b4befc60
Remove code deprecated in 2.12.0 ( #7126 )
...
* Remove smaller things.
* Remove from_settings().
* Remove old MediaPipeline init code.
* Cleanup.
* Typo.
2025-10-27 17:14:38 +05:00
d414d393d4
Remove Python 3.9 support ( #7121 )
2025-10-27 12:37:49 +01:00
eb496470f1
Add missing doc deps for new examples ( #7116 )
2025-10-24 11:58:18 +02:00
a5bbeb2586
Expanded documentation for issue #6676 ( #6957 )
2025-10-24 10:30:33 +02:00
2d073a9c0d
Fix typo in logging documentation ( #7109 )
2025-10-20 15:52:46 +02:00
b49aa2fb0c
Deprecate scrapy.core.downloader.handlers.http. ( #7079 )
2025-10-16 13:15:07 +02:00
a02abdcf63
Fix typos in documentation ( #7076 )
2025-10-06 11:43:36 +02:00
1c6ba00fd0
Move LogCounterHandler into a new extension ( #7046 )
...
* Move LogCounterHandler into Crawler.crawl().
* Add the LogCount extension.
* Update test_spider_custom_settings_log_level().
* Typo.
* Update docs.
2025-10-03 20:14:11 +05:00
f2531808f3
docs: Add Edit on GitHub ( #7058 )
2025-09-28 13:16:47 +05:00
2ad81a0ef8
Deprecate the spider arg of downloader middleware methods. ( #7037 )
...
* Deprecate the spider arg of downloader middleware methods.
* Add a test for deprecated spider args in downloader mws.
2025-09-02 21:51:47 +05:00
ed63fa94d6
Deprecate spider args of middlewares and pipelines. ( #7006 )
...
* Deprecate spider args of spider middleware methods.
* Deprecate the spider arg of pipeline process_item().
* Fix a typing issue.
* Deprecate the spider arg of pipeline {open,close}_spider().
* Cleanup.
* Update docs.
* Add pragma: no cover to some deprecated code.
* More tests.
* More tests.
* More tests.
* Simplify _process_parallel().
2025-08-21 13:23:10 +05:00
b68330811b
Deprecate spider arguments of StatsCollector methods ( #7011 )
2025-08-20 09:52:22 +02:00
edba1ad572
Refactor RobotsTxtMiddleware to use async def process_request(). ( #6802 )
2025-08-18 18:40:46 +02:00
89f53f0555
Added download_async() to ExecutionEngine and test cases for download() and download_async() ( #6842 )
...
* added async_download()
* add case tests for ExecutionEngine download() and async_download()
* change async_download() to download_async() in Engine class and subsequent tests
* fixed reactor import in test_engine.py
* Simplify TestEngineDownloadAsync.
* replaced tearDown by self.engine.downloader.close in test_engine.py and replace @defer.inlineCallbacks by @inlineCallbacks
* Simplify ExecutionEngine.download().
* Remove an unneeded import.
* Refactor ExecutionEngine.download{,_async}() tests.
* Deprecate ExecutionEngine.download().
* Update doc examples to use download_async().
* Replace uses of download() with download_async().
* Fix test_downloadermiddleware_robotstxt.py.
* Remove a stray file.
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-07-27 23:17:51 +05:00
552f2fb91e
Generic improvements related to reactorless tests ( #6968 )
...
* Skip doctests that import the reactor.
* Extract FTPDownloadHandler tests.
* Skip TestHttps2ClientProtocol via pytestmark.
* Replace some explicit reactor.callLater() in tests.
* Simplify TestRequestSendOrder._test_request_order().
2025-07-25 13:35:00 +05:00
d9ef0350d8
Merge remote-tracking branch 'origin/2.13'
2025-07-02 20:46:02 +05:00
155a504f24
Bump version: 2.13.2 → 2.13.3
2025-07-02 20:39:46 +05:00
cf465bf644
Release notes for 2.13.3. ( #6934 )
...
* Release notes for 2.13.3.
* Address feedback.
2025-07-02 20:36:01 +05:00
dabdc7550e
Deprecate CONCURRENT_REQUESTS_PER_IP setting ( #6921 )
...
* Deprecate CONCURRENT_REQUESTS_PER_IP setting
* Typing fix
* PR review
* tests
* oops
* Update scrapy/settings/default_settings.py
Co-authored-by: Adrián Chaves <adrian@chaves.gal>
* update logic
---------
Co-authored-by: Adrián Chaves <adrian@chaves.gal>
2025-06-30 16:51:05 +05:00
3843091c5f
Mark abstract base classes as such. ( #6930 )
...
* Mark abstract base classes as such.
* Remove an unneeded method.
* Fix exporter test coverage.
2025-06-30 15:14:34 +05:00
03fe7a6424
Add a deprecation notice for the offsite spider middleware.
2025-06-29 02:35:48 +05:00
2c1c10e923
Remove deprecated offsite spider middleware. ( #6926 )
...
* Remove deprecated offsite spider middleware.
* Add a deprecation notice for the offsite spider middleware.
2025-06-29 02:35:20 +05:00
ba1cedee1b
Document the new defaults of throttling settings ( #6923 )
2025-06-28 18:13:54 +05:00
3019393686
Document the new defaults of throttling settings ( #6923 )
2025-06-28 18:04:40 +05:00
1473f4d347
Remove the deprecated sphinx-hoverxref ( #6922 )
2025-06-28 18:04:06 +05:00
95172659af
Add sphinx-lint. ( #6920 )
2025-06-28 01:37:20 +02:00
843ad1afb1
Replace Black with Ruff in contributing.rst. ( #6903 )
2025-06-23 22:25:56 +05:00
712e965dbd
Replace Black with Ruff in contributing.rst. ( #6903 )
2025-06-23 17:56:48 +02:00
daa1a7d0b6
Remove the chdir fixture, re-enable fancy pytest asserts ( #6888 )
...
* Remove the chdir fixture.
* Re-enable fancy pytest asserts.
* Remove doc files from pytest ignores.
* Restore docs/_ext in test collection ignores.
* Skip a doctest that fails on Windows.
* Fix tests that were writing to the current dir.
2025-06-14 14:01:20 +05:00
0390176ecd
Merge branch '2.13'
2025-06-09 15:54:35 +05:00
c6740604a4
Bump version: 2.13.1 → 2.13.2
2025-06-09 15:21:06 +05:00
7400868ad5
Release notes for 2.13.2. ( #6868 )
...
* Release notes for 2.13.2.
* Update release notes.
2025-06-09 15:19:02 +05:00
405d9bc8a2
More docs for the is_asyncio_reactor_installed() behavior change. ( #6866 )
2025-06-06 12:59:49 +02:00
ajbmachon
Adham 3llam
Andrey Rakhmatullin
Thalison Fernandes
Adrián Chaves
Michael
sstegmanbsd
Sebastien Dionne
Laerte Pereira
Felipe Benevolo