From 71df815fcdbb294ac8fb6c4fe244eb05c737a310 Mon Sep 17 00:00:00 2001 From: ajbmachon Date: Wed, 31 Dec 2025 13:23:50 +0100 Subject: [PATCH] Clarify spider middleware handling of callback requests Make explicit that process_spider_output() handles both items and requests from callbacks before they reach pipelines or scheduler. --- docs/topics/lifecycle.rst | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/topics/lifecycle.rst b/docs/topics/lifecycle.rst index 179682562..1cd28ad8a 100644 --- a/docs/topics/lifecycle.rst +++ b/docs/topics/lifecycle.rst @@ -229,14 +229,16 @@ Spider middlewares process data at two points in the lifecycle: 1. **Start processing**: Via ``process_start()`` before initial requests reach the scheduler (see :ref:`lifecycle-start-processing`) -2. **Callback processing**: Via ``process_spider_input()`` and - ``process_spider_output()`` for responses and callback output +2. **Callback processing**: Via ``process_spider_input()`` before the callback + runs, and ``process_spider_output()`` after—processing both items and + new requests before they reach pipelines or the scheduler Common uses for callback processing include: - Filtering responses (e.g., by HTTP status code or content type) - Handling spider exceptions -- Modifying the items or requests yielded by callbacks +- Modifying or filtering items before they reach pipelines +- Modifying or filtering requests before they return to the scheduler For details on writing spider middlewares, see :ref:`topics-spider-middleware`.