71df815fcd
Clarify spider middleware handling of callback requests
...
Make explicit that process_spider_output() handles both items and
requests from callbacks before they reach pipelines or scheduler.
2025-12-31 13:23:50 +01:00
4e175e7930
Add spider middleware start request processing to lifecycle docs
...
Address reviewer feedback that spider middleware coverage was incomplete.
Changes include:
- Add new "Start request processing" section explaining process_start()
- Update lifecycle overview to include start processing as phase 2
- Update spider middleware section to show both integration points
- Update sequence diagram to show START PHASE and CRAWL PHASE
- Add missing signal mentions (engine_started, spider_opened,
request_dropped, spider_closed)
2025-12-31 13:20:45 +01:00
5941413af2
Add lifecycle documentation to toctree
...
Fixes sphinx warning about orphaned document by adding lifecycle.rst
to the Extending Scrapy section of the documentation index.
2025-12-30 08:35:22 +01:00
acabf82bde
Add Request to Response lifecycle documentation
...
Document the end-to-end flow of requests through Scrapy's internals,
including the engine's orchestration role, scheduling, downloading,
spider callback execution, and item pipeline processing.
This conceptual guide complements the existing architecture overview
by providing a detailed narrative of the request lifecycle.
2025-12-29 15:16:42 +01:00
9381ad893d
Remove outdated version marker directives (< 2.8.0) from documentation ( #7198 )
2025-12-22 10:00:39 +01:00
180ca39b23
Deprecate returning Deferreds from pipeline methods ( #7179 )
...
* Add tests for exceptions in pipelines.
* Deprecate returning Deferreds from pipeline process_item().
* Deprecate returning Deferreds from pipeline {open,close}_spider().
* Update the custom pipeline docs.
2025-12-16 13:35:07 +05:00
5a7e132486
Restore and deprecate the spider arg of Downloader._get_slot(). ( #7178 )
2025-12-15 11:42:24 +01:00
c49ae2115a
Refactor MediaPipeline.process_item() to async def ( #7177 )
2025-12-15 10:02:04 +01:00
588f3d4f65
Remove docs mentioning deprecated spider attrs. ( #7176 )
2025-12-15 09:51:56 +01:00
d8583a89c7
Change default priority queue to DownloaderAwarePriorityQueue ( #6940 )
...
* Change default priority queue to DownloaderAwarePriorityQueue
* Fix documentation building
* Simplify test_start_already_running_exception changes.
* Modernize the test.
* Fix TestEngineCloseSpider.
* Fix typing.
* Remove special slot=None handling.
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-12-11 11:25:17 +01:00
1a3e343dc4
Update conftest.py to refactor autouse pytest.skip() fixtures into pytest_runtest_setup() ( #7173 )
2025-12-11 09:48:25 +01:00
6ba6b032ad
Fix handling meta["download_slot"] == None. ( #7172 )
2025-12-11 09:46:51 +01:00
9bfa58e36c
Deprecate send_catch_log_deferred(). ( #7161 )
...
* Add a test for not having pending tasks.
* Refactor TestFeedExporterSignals.
* Refactor FeedExporter.close_spider().
* More engine start/stop robustness.
* Refactor send_catch_log_async(), deprecate send_catch_log_deferred().
* Add pragma: no cover.
* Warn on signal handlers returning a Deferred.
* Make _pending_close_coros an instance attribute.
* Remove an unused function.
* Remove the outdated TODO.
2025-12-10 14:42:49 +05:00
5105f55a98
Add `DownloaderMiddlewareManager.download_async()`, deprecate returning Deferreds from downloader mws ( #7069 )
...
* Restore coverage in MiddlewareManager spider arg handling.
* Move MiddlewareManager._warn_spider_arg() tests.
* Add DownloaderMiddlewareManager.download_async().
* Make download_func async.
* Inline MiddlewareManager._warn_spider_arg().
* Deprecate returning deferreds from downloader middlewares.
* Update the downloader mw doc about async def.
* Use global_object_name() instead of repr().
* Update docs/topics/downloader-middleware.rst
Co-authored-by: Adrián Chaves <adrian@chaves.io>
* Update downloader-middleware.rst
---------
Co-authored-by: Adrián Chaves <adrian@chaves.io>
2025-12-09 20:07:36 +05:00
7ed20ee7f3
Limit the queue size in _parallel_asyncio(). ( #7159 )
2025-12-09 15:24:40 +01:00
483e059d59
Deprecate returning Deferreds from spider callbacks and errbacks. ( #7151 )
2025-12-09 15:03:32 +01:00
11073c8680
Deprecate spider attributes that can be replaced by settings, round 2 ( #7039 )
...
* Move duplicate code to utils
* move new function to end
* draft
* update docs
* Update tests
* Update test name
* rename test
* rollback signature
* leftover
* sort
* Rollback some changes
* Rollback download_delay warning
* Rollback
* fix checks
* Remove unused imports
* Add pragma: no cover.
---------
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-12-01 01:26:17 +05:00
1e8de24380
Merge pull request #7145 from Icorebleidd/master
...
Fix duplicate assertion in test_utils_deprecate
2025-11-28 01:16:02 +05:00
0adc561348
Add a list of remaining Deferred APIs. ( #7146 )
2025-11-27 20:13:04 +05:00
813fd9f1ac
Restore brotlicffi support ( #7160 )
...
* restore brotlicffi dependency
* pre-commit
2025-11-25 12:06:55 +05:00
4cb0144b39
Merge tag '2.13.4'
...
Bump version: 2.13.3 → 2.13.4
2025-11-17 16:47:54 +05:00
2f62ab532d
Bump version: 2.13.3 → 2.13.4
2025-11-17 16:30:37 +05:00
31a9c03c24
Release notes for 2.13.4. ( #7144 )
...
* Release notes for 2.13.4.
* Formatting.
2025-11-17 16:30:04 +05:00
c44b8df6c7
Cherry-pick: Mitigate brotli and deflate decompression bombs DoS ( #7134 )
...
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-11-17 16:25:00 +05:00
14737e91ed
Mitigate brotli and deflate decompression bombs DoS ( #7134 )
...
Co-authored-by: Eugenio Lacuesta <eugenio.lacuesta@gmail.com>
Co-authored-by: Andrey Rakhmatullin <wrar@wrar.name>
2025-11-17 14:21:50 +05:00
d091256c58
Remove the deprecated sphinx-hoverxref ( #6922 )
2025-11-14 15:39:21 +05:00
c83ca70db3
Don't force the unavailable sysmon coverage core. ( #7137 )
2025-11-13 23:26:08 +05:00
85e4e6c42b
Pin Twisted to <= 25.5.0 due to internal API changes.
2025-11-13 23:25:51 +05:00
426aafddca
Don't force the unavailable sysmon coverage core. ( #7137 )
2025-11-13 23:22:52 +05:00
db37040a09
Merge pull request #7142 from wRAR/pin-twisted
...
Pin Twisted to <= 25.5.0 due to internal API changes.
2025-11-13 22:48:33 +05:00
ba30e8b82c
Pin Twisted to <= 25.5.0 due to internal API changes.
2025-11-10 00:58:27 +05:00
8c5fa6e6ae
Line buffering ( #7094 )
2025-10-27 15:42:43 +01:00
804ae167df
Update tool versions ( #7127 )
2025-10-27 14:11:31 +01:00
61b4befc60
Remove code deprecated in 2.12.0 ( #7126 )
...
* Remove smaller things.
* Remove from_settings().
* Remove old MediaPipeline init code.
* Cleanup.
* Typo.
2025-10-27 17:14:38 +05:00
d414d393d4
Remove Python 3.9 support ( #7121 )
2025-10-27 12:37:49 +01:00
e48a1bdde3
Speed up and refactor command tests ( #7118 )
2025-10-27 12:28:16 +01:00
9cce6c30db
Undeprecate download_delay. ( #7117 )
2025-10-24 17:44:43 +05:00
5afc9b0221
Automatically transpose images using EXIF data ( #6975 )
...
* Automatically transpose images using EXIF data
* Added test
---------
Co-authored-by: Andrew Murray <radarhere@users.noreply.github.com>
2025-10-24 16:08:23 +05:00
eb496470f1
Add missing doc deps for new examples ( #7116 )
2025-10-24 11:58:18 +02:00
a5bbeb2586
Expanded documentation for issue #6676 ( #6957 )
2025-10-24 10:30:33 +02:00
2d073a9c0d
Fix typo in logging documentation ( #7109 )
2025-10-20 15:52:46 +02:00
e98c1644ce
Add unit tests that check download handlers behave like basic http clients ( #6946 ) ( #7059 )
2025-10-20 13:08:26 +02:00
b49aa2fb0c
Deprecate scrapy.core.downloader.handlers.http. ( #7079 )
2025-10-16 13:15:07 +02:00
7b215c6578
UTF-8 BOM at the beginning of the file ignored ( #7095 )
2025-10-15 18:15:07 +04:00
4865a500b6
resolves #6876 ( #6977 )
2025-10-09 22:14:34 +02:00
a02abdcf63
Fix typos in documentation ( #7076 )
2025-10-06 11:43:36 +02:00
c577771838
Drop tests/upper-constraints.txt. ( #7073 )
2025-10-06 11:32:20 +02:00
10850e7d29
Fix extra stdout output in tests ( #7070 )
2025-10-06 11:19:34 +02:00
1c6ba00fd0
Move LogCounterHandler into a new extension ( #7046 )
...
* Move LogCounterHandler into Crawler.crawl().
* Add the LogCount extension.
* Update test_spider_custom_settings_log_level().
* Typo.
* Update docs.
2025-10-03 20:14:11 +05:00
798390c096
Improve performance of deprecated spider arg checks. ( #7045 )
2025-10-03 14:17:46 +02:00
ajbmachon
Adham 3llam
Andrey Rakhmatullin
Thalison Fernandes
DenisC
Laerte Pereira
Leonardo Tozzo
Rui Xi
Adrián Chaves
Mikhail Korobov
Vasiliy Kiryanov
Andrew Murray
Michael
sstegmanbsd
Wes Galbraith
Vasiliy Kiryanov
adglee
Sebastien Dionne