diff --git a/docs/index.rst b/docs/index.rst index 688cab81b..8e0449e82 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -223,6 +223,7 @@ Extending Scrapy :hidden: topics/architecture + topics/lifecycle topics/addons topics/downloader-middleware topics/spider-middleware @@ -238,6 +239,9 @@ Extending Scrapy :doc:`topics/architecture` Understand the Scrapy architecture. +:doc:`topics/lifecycle` + Follow a request from creation to response delivery. + :doc:`topics/addons` Enable and configure third-party extensions. diff --git a/docs/topics/lifecycle.rst b/docs/topics/lifecycle.rst new file mode 100644 index 000000000..1cd28ad8a --- /dev/null +++ b/docs/topics/lifecycle.rst @@ -0,0 +1,436 @@ +.. _topics-lifecycle: + +============================= +Request to Response Lifecycle +============================= + +This document explains how a :class:`~scrapy.Request` flows through Scrapy's +internals, from creation in a spider to the delivery of a +:class:`~scrapy.http.Response` back to a spider callback. Understanding this +lifecycle helps when debugging, optimizing performance, or extending Scrapy +with custom components. + +For a high-level component overview, see :ref:`topics-architecture`. This page +focuses on the detailed sequence of operations. + +.. _lifecycle-overview: + +Lifecycle overview +================== + +A request passes through these main phases: + +1. **Creation**: A spider yields requests and items from its :meth:`~scrapy.Spider.start` method +2. **Start processing**: Spider middleware processes start output via :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start` +3. **Scheduling**: The engine passes the request to the scheduler for queuing +4. **Downloading**: The scheduler returns the request to the engine, which sends it to the downloader +5. **Response handling**: The downloader returns a response to the engine +6. **Spider processing**: The engine passes the response to the spider for callback execution +7. **Output processing**: Items go to pipelines; new requests return to step 3 + +The following sections describe each phase in detail. + +.. _lifecycle-engine-role: + +The engine as orchestrator +========================== + +The :ref:`execution engine ` controls all data flow between +Scrapy components. It does not process requests or responses itself; instead, +it coordinates when each component acts and manages the transitions between +phases. + +The engine's responsibilities include: + +- Obtaining start requests from the spider and passing them to the scheduler +- Requesting the next request from the scheduler when capacity is available +- Sending requests to the downloader and receiving responses +- Passing responses to the scraper for spider callback execution +- Routing callback output (items and new requests) to the appropriate components +- Monitoring idle conditions and initiating spider closure + +The engine implements backpressure by checking whether the downloader or +scraper can accept more work before dequeuing additional requests from the +scheduler. This prevents memory exhaustion when spiders generate requests +faster than they can be processed. + +**Backpressure conditions** + +The engine pauses request processing when any of these conditions are true: + +- The downloader has reached its concurrency limit (:setting:`CONCURRENT_REQUESTS`) +- The scraper's active response size exceeds its threshold (:setting:`SCRAPER_SLOT_MAX_ACTIVE_SIZE`) +- The engine is shutting down + +When the engine starts, it emits the :signal:`engine_started` signal. + +.. _lifecycle-start-processing: + +Start request processing +======================== + +Before any crawling begins, the spider's :meth:`~scrapy.Spider.start` method +(or the deprecated :meth:`~scrapy.Spider.start_requests`) generates the initial +requests and items. This output passes through :ref:`spider middleware +` before requests reach the scheduler. + +**Start processing flow** + +1. The engine opens the spider and emits the :signal:`spider_opened` signal +2. The engine calls the spider middleware manager's ``process_start()`` method +3. Each spider middleware's :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start` + method can filter, transform, or replace the start output +4. Requests from the processed output are passed to the scheduler +5. Items from the processed output are sent directly to pipelines + +Spider middlewares can use ``process_start()`` to: + +- Filter out certain start requests based on custom logic +- Add metadata to requests (e.g., setting ``request.meta`` values) +- Transform URLs or request parameters +- Inject additional requests not in the original start output + +For details on implementing ``process_start()``, see +:ref:`topics-spider-middleware`. + +.. _lifecycle-scheduling: + +Request scheduling and duplicate filtering +========================================== + +When the engine receives a request (from start processing or spider callbacks), +it passes the request to the :ref:`scheduler `. + +**Scheduling process** + +1. The engine emits the :signal:`request_scheduled` signal +2. If a signal handler raises :exc:`~scrapy.exceptions.IgnoreRequest`, the request is dropped +3. The scheduler checks for duplicates using the configured duplicate filter +4. If the request is a duplicate (and ``dont_filter=False``), it is rejected and the :signal:`request_dropped` signal is emitted +5. Otherwise, the request is added to the scheduler's queue + +**Duplicate filtering** + +The default duplicate filter (:class:`~scrapy.dupefilters.RFPDupeFilter`) uses +request fingerprints to identify duplicates. A fingerprint is computed from the +request's URL, method, and body. The filter maintains a set of seen +fingerprints and rejects requests whose fingerprint already exists. + +To bypass duplicate filtering for a specific request, set ``dont_filter=True`` +when creating the request:: + + yield scrapy.Request(url, dont_filter=True) + +For custom duplicate filtering logic, implement a class following the +:class:`~scrapy.dupefilters.BaseDupeFilter` interface and configure it via +the :setting:`DUPEFILTER_CLASS` setting. + +**Queue structure** + +The default scheduler maintains two queues: + +- **Memory queue**: Stores requests in memory for fast access +- **Disk queue**: Persists requests to disk when a job directory is configured + +When dequeuing, the scheduler checks the memory queue first, then falls back to +the disk queue. This design supports :ref:`pausing and resuming crawls +`. + +For more details on the scheduler, see :ref:`topics-scheduler`. + +.. _lifecycle-downloading: + +Downloading +=========== + +When the engine determines it has capacity for more downloads, it requests +the next request from the scheduler and passes it to the +:ref:`downloader `. + +**Download process** + +1. The engine calls the downloader with the request +2. The request passes through the :ref:`downloader middleware chain ` (``process_request`` methods) +3. If no middleware returns a response, the request reaches a download handler +4. The download handler performs the actual HTTP request +5. The response passes back through the downloader middleware chain (``process_response`` methods) +6. The final response returns to the engine + +**Downloader middleware integration** + +Downloader middlewares can intercept requests before they reach the network +and responses before they reach the spider. Common uses include: + +- Setting headers (User-Agent, cookies, authentication) +- Handling redirects and retries +- Caching responses +- Returning synthetic responses without making network requests + +Each middleware's ``process_request`` method can: + +- Return ``None`` to continue to the next middleware +- Return a :class:`~scrapy.http.Response` to skip remaining middlewares and the download handler +- Return a :class:`~scrapy.Request` to reschedule a different request +- Raise :exc:`~scrapy.exceptions.IgnoreRequest` to abort the request + +For details on writing downloader middlewares, see +:ref:`topics-downloader-middleware`. + +**Concurrency and delays** + +The downloader enforces concurrency limits at two levels: + +- **Global**: :setting:`CONCURRENT_REQUESTS` limits total simultaneous downloads +- **Per-domain or per-IP**: :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` or :setting:`CONCURRENT_REQUESTS_PER_IP` limit downloads to each target + +Download delays can be configured via :setting:`DOWNLOAD_DELAY`. When set, the +downloader waits at least this many seconds between consecutive requests to +the same domain. The :setting:`RANDOMIZE_DOWNLOAD_DELAY` setting adds +randomization to make request timing less predictable. + +**Signals emitted during download** + +- :signal:`request_reached_downloader`: When a request enters the downloader's active set +- :signal:`response_downloaded`: When the download handler returns a response +- :signal:`request_left_downloader`: When processing for a request completes +- :signal:`bytes_received`: When data chunks arrive during download +- :signal:`headers_received`: When HTTP headers are received + +.. _lifecycle-spider-processing: + +Spider callback execution +========================= + +After the engine receives a response from the downloader, it passes the +response to the scraper, which manages spider callback execution. + +**Callback execution process** + +1. The response enters the scraper's queue +2. The response passes through :ref:`spider middleware ` (``process_spider_input`` methods) +3. The spider's callback method is invoked with the response +4. The callback's output (an iterable of items and requests) passes through spider middleware (``process_spider_output`` methods) +5. Items and requests are extracted from the processed output + +**Callback selection** + +The callback is determined by the request that generated the response: + +- If ``request.callback`` is set, that function is called +- Otherwise, the spider's default ``_parse`` method (which calls ``parse``) is used + +If the download resulted in an error and the request has an ``errback``, that +function is called instead with a :class:`~twisted.python.failure.Failure` +object. + +**Spider middleware integration** + +Spider middlewares process data at two points in the lifecycle: + +1. **Start processing**: Via ``process_start()`` before initial requests reach + the scheduler (see :ref:`lifecycle-start-processing`) +2. **Callback processing**: Via ``process_spider_input()`` before the callback + runs, and ``process_spider_output()`` after—processing both items and + new requests before they reach pipelines or the scheduler + +Common uses for callback processing include: + +- Filtering responses (e.g., by HTTP status code or content type) +- Handling spider exceptions +- Modifying or filtering items before they reach pipelines +- Modifying or filtering requests before they return to the scheduler + +For details on writing spider middlewares, see :ref:`topics-spider-middleware`. + +**Signals emitted during spider processing** + +- :signal:`response_received`: When the engine receives a response (before spider processing) +- :signal:`spider_error`: When a spider callback raises an exception + +.. _lifecycle-item-processing: + +Item pipeline processing +======================== + +When a spider callback yields an item (a dict, :class:`~scrapy.Item`, or +dataclass), the scraper passes it to the :ref:`item pipeline +`. + +**Pipeline execution** + +1. The item passes to the first pipeline's ``process_item`` method +2. If the pipeline returns an item, it passes to the next pipeline +3. This continues until all pipelines have processed the item +4. If any pipeline raises :exc:`~scrapy.exceptions.DropItem`, processing stops + +**Pipeline configuration** + +Pipelines are enabled via the :setting:`ITEM_PIPELINES` setting, which maps +pipeline classes to integer priority values. Lower values execute first. + +**Signals emitted during item processing** + +- :signal:`item_scraped`: When an item successfully passes through all pipelines +- :signal:`item_dropped`: When a pipeline raises :exc:`~scrapy.exceptions.DropItem` +- :signal:`item_error`: When a pipeline raises an unexpected exception + +For details on writing item pipelines, see :ref:`topics-item-pipeline`. + +.. _lifecycle-new-requests: + +New request handling +==================== + +When a spider callback yields a :class:`~scrapy.Request`, the scraper extracts +it from the callback output and passes it back to the engine. The engine then +schedules the request, and the lifecycle repeats from the scheduling phase. + +This recursive flow continues until: + +- The scheduler has no pending requests +- All active downloads have completed +- The spider's start iterator is exhausted +- The scraper has no active responses + +When all these conditions are met, the spider is considered idle. + +.. _lifecycle-error-handling: + +Error handling +============== + +Scrapy handles errors at multiple points in the lifecycle. + +**Download errors** + +When a download fails (network error, timeout, etc.): + +1. The error passes through downloader middleware ``process_exception`` methods +2. If a middleware returns a response or request, normal processing continues +3. Otherwise, if the request has an ``errback``, it is called with the failure +4. If no ``errback`` exists, the error is logged + +The :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware` handles +retries for failed requests. It reschedules requests that fail due to +connection errors, timeouts, or certain HTTP status codes, up to a +configurable limit (:setting:`RETRY_TIMES`). + +**Spider callback errors** + +When a spider callback raises an exception: + +1. The error passes through spider middleware ``process_spider_exception`` methods +2. If a middleware yields items or requests, those are processed normally +3. The :signal:`spider_error` signal is emitted +4. If the exception is :exc:`~scrapy.exceptions.CloseSpider`, the spider shuts down + +**Item pipeline errors** + +When a pipeline's ``process_item`` raises an exception: + +- :exc:`~scrapy.exceptions.DropItem`: The item is dropped (normal behavior) +- Other exceptions: The :signal:`item_error` signal is emitted and the error is logged + +.. _lifecycle-spider-idle: + +Spider idle and closure +======================= + +The engine periodically checks whether the spider is idle. A spider is +considered idle when: + +- The scraper has no responses being processed +- The downloader has no active requests +- The start request iterator is exhausted +- The scheduler has no pending requests + +When the spider becomes idle: + +1. The engine emits the :signal:`spider_idle` signal +2. Signal handlers can schedule new requests to keep the spider running +3. If a handler raises :exc:`~scrapy.exceptions.DontCloseSpider`, the spider remains open +4. Otherwise, the engine initiates spider closure and emits the :signal:`spider_closed` signal + +The closure reason is "finished" by default, but can be customized by raising +:exc:`~scrapy.exceptions.CloseSpider` with a reason argument. + +.. _lifecycle-customization: + +Customization points +==================== + +This section summarizes where you can customize the request lifecycle. + +**Component replacement** + +These settings allow replacing core components with custom implementations: + +- :setting:`SCHEDULER`: Custom scheduler class +- :setting:`DUPEFILTER_CLASS`: Custom duplicate filter +- :setting:`DOWNLOADER`: Custom downloader class + +**Middleware chains** + +These settings configure middleware that processes requests and responses: + +- :setting:`DOWNLOADER_MIDDLEWARES`: Modify requests before download and responses after +- :setting:`SPIDER_MIDDLEWARES`: Process responses before callbacks and output after + +**Pipeline chain** + +- :setting:`ITEM_PIPELINES`: Process items after extraction + +**Signal handlers** + +:ref:`Signals ` allow reacting to lifecycle events without +modifying core components. Extensions typically connect to signals to implement +cross-cutting functionality. + +**Per-request customization** + +Individual requests support these customization options: + +- ``callback``: Function to process the response +- ``errback``: Function to handle download errors +- ``dont_filter``: Skip duplicate filtering +- ``priority``: Influence dequeue order in the scheduler +- ``meta``: Pass data between middlewares and callbacks + +For the complete request API, see :ref:`topics-request-response`. + +.. _lifecycle-diagram: + +Lifecycle sequence diagram +========================== + +The following diagram illustrates the request lifecycle:: + + Spider Engine Scheduler Downloader Scraper + | | | | | + | | | | | + |=== START PHASE ================================================| + | | | | | + |--start()----->| | | | + | [Spider MW: process_start()] | | | + | |---enqueue----->| | | + | | | | | + |=== CRAWL PHASE (repeats) ======================================| + | | | | | + | |<--next_request-| | | + | | | | | + | |----Request-----|-------------->| | + | | | [Downloader Middlewares] | + | | | [Download Handler] | + | | | | | + | |<---Response----|---------------| | + | | | | | + | |----Response----|---------------|------------->| + | | | | | + | | | | [Spider MW] | + | | | | [Callback] | + |<--items,reqs--|----------------|---------------|--------------| + | | | | | + | | [Items to Pipeline, Requests to Scheduler] | + | | | | | + +The crawl phase repeats until the spider is idle and no handlers prevent closure.