RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/docs/topics/lifecycle.rst

15 KiB
Raw Blame History

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> </head>

Request to Response Lifecycle

This document explains how a :class:`~scrapy.Request` flows through Scrapy's internals, from creation in a spider to the delivery of a :class:`~scrapy.http.Response` back to a spider callback. Understanding this lifecycle helps when debugging, optimizing performance, or extending Scrapy with custom components.

System Message: ERROR/3 (<stdin>, line 7); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 7); backlink

Unknown interpreted text role "class".

For a high-level component overview, see :ref:`topics-architecture`. This page focuses on the detailed sequence of operations.

System Message: ERROR/3 (<stdin>, line 13); backlink

Unknown interpreted text role "ref".

Lifecycle overview

A request passes through these main phases:

  1. Creation: A spider yields a :class:`~scrapy.Request`

    System Message: ERROR/3 (<stdin>, line 23); backlink

    Unknown interpreted text role "class".

  2. Scheduling: The engine passes the request to the scheduler for queuing

  3. Downloading: The scheduler returns the request to the engine, which sends it to the downloader

  4. Response handling: The downloader returns a response to the engine

  5. Spider processing: The engine passes the response to the spider for callback execution

  6. Output processing: Items go to pipelines; new requests return to step 2

The following sections describe each phase in detail.

The engine as orchestrator

The :ref:`execution engine <component-engine>` controls all data flow between Scrapy components. It does not process requests or responses itself; instead, it coordinates when each component acts and manages the transitions between phases.

System Message: ERROR/3 (<stdin>, line 37); backlink

Unknown interpreted text role "ref".

The engine's responsibilities include:

  • Obtaining start requests from the spider and passing them to the scheduler
  • Requesting the next request from the scheduler when capacity is available
  • Sending requests to the downloader and receiving responses
  • Passing responses to the scraper for spider callback execution
  • Routing callback output (items and new requests) to the appropriate components
  • Monitoring idle conditions and initiating spider closure

The engine implements backpressure by checking whether the downloader or scraper can accept more work before dequeuing additional requests from the scheduler. This prevents memory exhaustion when spiders generate requests faster than they can be processed.

Backpressure conditions

The engine pauses request processing when any of these conditions are true:

Request scheduling and duplicate filtering

When the engine receives a request (from start requests or spider callbacks), it passes the request to the :ref:`scheduler <component-scheduler>`.

System Message: ERROR/3 (<stdin>, line 69); backlink

Unknown interpreted text role "ref".

Scheduling process

  1. The engine emits the :signal:`request_scheduled` signal

    System Message: ERROR/3 (<stdin>, line 74); backlink

    Unknown interpreted text role "signal".

  2. If a signal handler raises :exc:`~scrapy.exceptions.IgnoreRequest`, the request is dropped

    System Message: ERROR/3 (<stdin>, line 75); backlink

    Unknown interpreted text role "exc".

  3. The scheduler checks for duplicates using the configured duplicate filter

  4. If the request is a duplicate (and dont_filter=False), it is rejected

  5. Otherwise, the request is added to the scheduler's queue

Duplicate filtering

The default duplicate filter (:class:`~scrapy.dupefilters.RFPDupeFilter`) uses request fingerprints to identify duplicates. A fingerprint is computed from the request's URL, method, and body. The filter maintains a set of seen fingerprints and rejects requests whose fingerprint already exists.

System Message: ERROR/3 (<stdin>, line 82); backlink

Unknown interpreted text role "class".

To bypass duplicate filtering for a specific request, set dont_filter=True when creating the request:

yield scrapy.Request(url, dont_filter=True)

For custom duplicate filtering logic, implement a class following the :class:`~scrapy.dupefilters.BaseDupeFilter` interface and configure it via the :setting:`DUPEFILTER_CLASS` setting.

System Message: ERROR/3 (<stdin>, line 92); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 92); backlink

Unknown interpreted text role "setting".

Queue structure

The default scheduler maintains two queues:

  • Memory queue: Stores requests in memory for fast access
  • Disk queue: Persists requests to disk when a job directory is configured

When dequeuing, the scheduler checks the memory queue first, then falls back to the disk queue. This design supports :ref:`pausing and resuming crawls <topics-jobs>`.

System Message: ERROR/3 (<stdin>, line 103); backlink

Unknown interpreted text role "ref".

For more details on the scheduler, see :ref:`topics-scheduler`.

System Message: ERROR/3 (<stdin>, line 107); backlink

Unknown interpreted text role "ref".

Downloading

When the engine determines it has capacity for more downloads, it requests the next request from the scheduler and passes it to the :ref:`downloader <component-downloader>`.

System Message: ERROR/3 (<stdin>, line 114); backlink

Unknown interpreted text role "ref".

Download process

  1. The engine calls the downloader with the request

  2. The request passes through the :ref:`downloader middleware chain <component-downloader-middleware>` (process_request methods)

    System Message: ERROR/3 (<stdin>, line 121); backlink

    Unknown interpreted text role "ref".

  3. If no middleware returns a response, the request reaches a download handler

  4. The download handler performs the actual HTTP request

  5. The response passes back through the downloader middleware chain (process_response methods)

  6. The final response returns to the engine

Downloader middleware integration

Downloader middlewares can intercept requests before they reach the network and responses before they reach the spider. Common uses include:

  • Setting headers (User-Agent, cookies, authentication)
  • Handling redirects and retries
  • Caching responses
  • Returning synthetic responses without making network requests

Each middleware's process_request method can:

For details on writing downloader middlewares, see :ref:`topics-downloader-middleware`.

System Message: ERROR/3 (<stdin>, line 144); backlink

Unknown interpreted text role "ref".

Concurrency and delays

The downloader enforces concurrency limits at two levels:

Download delays can be configured via :setting:`DOWNLOAD_DELAY`. When set, the downloader waits at least this many seconds between consecutive requests to the same domain. The :setting:`RANDOMIZE_DOWNLOAD_DELAY` setting adds randomization to make request timing less predictable.

System Message: ERROR/3 (<stdin>, line 154); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 154); backlink

Unknown interpreted text role "setting".

Signals emitted during download

Spider callback execution

After the engine receives a response from the downloader, it passes the response to the scraper, which manages spider callback execution.

Callback execution process

  1. The response enters the scraper's queue

  2. The response passes through :ref:`spider middleware <component-spider-middleware>` (process_spider_input methods)

    System Message: ERROR/3 (<stdin>, line 178); backlink

    Unknown interpreted text role "ref".

  3. The spider's callback method is invoked with the response

  4. The callback's output (an iterable of items and requests) passes through spider middleware (process_spider_output methods)

  5. Items and requests are extracted from the processed output

Callback selection

The callback is determined by the request that generated the response:

  • If request.callback is set, that function is called
  • Otherwise, the spider's default _parse method (which calls parse) is used

If the download resulted in an error and the request has an errback, that function is called instead with a :class:`~twisted.python.failure.Failure` object.

System Message: ERROR/3 (<stdin>, line 190); backlink

Unknown interpreted text role "class".

Spider middleware integration

Spider middlewares can process responses before they reach the callback and filter or transform the callback's output. Common uses include:

  • Filtering responses (e.g., by HTTP status code or content type)
  • Handling spider exceptions
  • Modifying the items or requests yielded by callbacks

For details on writing spider middlewares, see :ref:`topics-spider-middleware`.

System Message: ERROR/3 (<stdin>, line 203); backlink

Unknown interpreted text role "ref".

Signals emitted during spider processing

  • :signal:`response_received`: When the engine receives a response (before spider processing)

    System Message: ERROR/3 (<stdin>, line 207); backlink

    Unknown interpreted text role "signal".

  • :signal:`spider_error`: When a spider callback raises an exception

    System Message: ERROR/3 (<stdin>, line 208); backlink

    Unknown interpreted text role "signal".

Item pipeline processing

When a spider callback yields an item (a dict, :class:`~scrapy.Item`, or dataclass), the scraper passes it to the :ref:`item pipeline <component-pipelines>`.

System Message: ERROR/3 (<stdin>, line 215); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 215); backlink

Unknown interpreted text role "ref".

Pipeline execution

  1. The item passes to the first pipeline's process_item method

  2. If the pipeline returns an item, it passes to the next pipeline

  3. This continues until all pipelines have processed the item

  4. If any pipeline raises :exc:`~scrapy.exceptions.DropItem`, processing stops

    System Message: ERROR/3 (<stdin>, line 224); backlink

    Unknown interpreted text role "exc".

Pipeline configuration

Pipelines are enabled via the :setting:`ITEM_PIPELINES` setting, which maps pipeline classes to integer priority values. Lower values execute first.

System Message: ERROR/3 (<stdin>, line 228); backlink

Unknown interpreted text role "setting".

Signals emitted during item processing

For details on writing item pipelines, see :ref:`topics-item-pipeline`.

System Message: ERROR/3 (<stdin>, line 237); backlink

Unknown interpreted text role "ref".

New request handling

When a spider callback yields a :class:`~scrapy.Request`, the scraper extracts it from the callback output and passes it back to the engine. The engine then schedules the request, and the lifecycle repeats from the scheduling phase.

System Message: ERROR/3 (<stdin>, line 244); backlink

Unknown interpreted text role "class".

This recursive flow continues until:

  • The scheduler has no pending requests
  • All active downloads have completed
  • The spider's start iterator is exhausted
  • The scraper has no active responses

When all these conditions are met, the spider is considered idle.

Error handling

Scrapy handles errors at multiple points in the lifecycle.

Download errors

When a download fails (network error, timeout, etc.):

  1. The error passes through downloader middleware process_exception methods
  2. If a middleware returns a response or request, normal processing continues
  3. Otherwise, if the request has an errback, it is called with the failure
  4. If no errback exists, the error is logged

The :class:`~scrapy.downloadermiddlewares.retry.RetryMiddleware` handles retries for failed requests. It reschedules requests that fail due to connection errors, timeouts, or certain HTTP status codes, up to a configurable limit (:setting:`RETRY_TIMES`).

System Message: ERROR/3 (<stdin>, line 273); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 273); backlink

Unknown interpreted text role "setting".

Spider callback errors

When a spider callback raises an exception:

  1. The error passes through spider middleware process_spider_exception methods

  2. If a middleware yields items or requests, those are processed normally

  3. The :signal:`spider_error` signal is emitted

    System Message: ERROR/3 (<stdin>, line 284); backlink

    Unknown interpreted text role "signal".

  4. If the exception is :exc:`~scrapy.exceptions.CloseSpider`, the spider shuts down

    System Message: ERROR/3 (<stdin>, line 285); backlink

    Unknown interpreted text role "exc".

Item pipeline errors

When a pipeline's process_item raises an exception:

Spider idle and closure

The engine periodically checks whether the spider is idle. A spider is considered idle when:

  • The scraper has no responses being processed
  • The downloader has no active requests
  • The start request iterator is exhausted
  • The scheduler has no pending requests

When the spider becomes idle:

  1. The engine emits the :signal:`spider_idle` signal

    System Message: ERROR/3 (<stdin>, line 309); backlink

    Unknown interpreted text role "signal".

  2. Signal handlers can schedule new requests to keep the spider running

  3. If a handler raises :exc:`~scrapy.exceptions.DontCloseSpider`, the spider remains open

    System Message: ERROR/3 (<stdin>, line 311); backlink

    Unknown interpreted text role "exc".

  4. Otherwise, the engine initiates spider closure

The closure reason is "finished" by default, but can be customized by raising :exc:`~scrapy.exceptions.CloseSpider` with a reason argument.

System Message: ERROR/3 (<stdin>, line 314); backlink

Unknown interpreted text role "exc".

Customization points

This section summarizes where you can customize the request lifecycle.

Component replacement

These settings allow replacing core components with custom implementations:

Middleware chains

These settings configure middleware that processes requests and responses:

Pipeline chain

Signal handlers

:ref:`Signals <topics-signals>` allow reacting to lifecycle events without modifying core components. Extensions typically connect to signals to implement cross-cutting functionality.

System Message: ERROR/3 (<stdin>, line 345); backlink

Unknown interpreted text role "ref".

Per-request customization

Individual requests support these customization options:

  • callback: Function to process the response
  • errback: Function to handle download errors
  • dont_filter: Skip duplicate filtering
  • priority: Influence dequeue order in the scheduler
  • meta: Pass data between middlewares and callbacks

For the complete request API, see :ref:`topics-request-response`.

System Message: ERROR/3 (<stdin>, line 359); backlink

Unknown interpreted text role "ref".

Lifecycle sequence diagram

The following diagram illustrates the request lifecycle:

Spider              Engine              Scheduler           Downloader          Scraper
  |                   |                    |                    |                  |
  |---Request-------->|                    |                    |                  |
  |                   |---enqueue--------->|                    |                  |
  |                   |                    |                    |                  |
  |                   |<--next_request-----|                    |                  |
  |                   |                    |                    |                  |
  |                   |-------Request------|----------------->  |                  |
  |                   |                    |    [Downloader Middlewares]           |
  |                   |                    |    [Download Handler]                 |
  |                   |                    |                    |                  |
  |                   |<------Response-----|-------------------|                   |
  |                   |                    |                    |                  |
  |                   |------Response------|------------------- |---------------->|
  |                   |                    |                    |                  |
  |                   |                    |                    | [Spider Middlewares]
  |                   |                    |                    | [Callback]       |
  |<---items,requests-|--------------------|------------------- |-----------------|
  |                   |                    |                    |                  |
  |                   |   [Items to Pipeline, Requests to Scheduler]              |
  |                   |                    |                    |                  |

This cycle repeats until the spider is idle and no handlers prevent closure.

</html>