From e0a7de7213e2f06a94917624bad572a41dcc3905 Mon Sep 17 00:00:00 2001 From: Labib Bin Salam <98468420+Labib-Bin-Salam@users.noreply.github.com> Date: Thu, 11 Jun 2026 10:40:15 +0100 Subject: [PATCH] Document cb_kwargs/meta deep-copy when JOBDIR is set (#7573) When JOBDIR is enabled, requests are serialized to disk with pickle, so the objects stored in a request's cb_kwargs and meta are deep-copied on the round trip. Callbacks then receive copies rather than the original objects, which is easy to miss and can silently break code that relies on sharing mutable state. Add a note to the request serialization section of the jobs docs and a cross-referenced caution to the Request.cb_kwargs attribute docs. Closes #6120 --- docs/topics/jobs.rst | 9 +++++++++ docs/topics/request-response.rst | 7 +++++++ 2 files changed, 16 insertions(+) diff --git a/docs/topics/jobs.rst b/docs/topics/jobs.rst index 769925dd5..8d6b2458b 100644 --- a/docs/topics/jobs.rst +++ b/docs/topics/jobs.rst @@ -104,6 +104,15 @@ If you wish to log the requests that couldn't be serialized, you can set the :setting:`SCHEDULER_DEBUG` setting to ``True`` in the project's settings page. It is ``False`` by default. +.. note:: Because requests are serialized with :mod:`pickle`, the objects you + store on a request, such as the values of its + :attr:`~scrapy.Request.cb_kwargs` and :attr:`~scrapy.Request.meta` + dictionaries, are deep-copied when the request is written to and later read + back from the job directory. As a result, the callback receives a *copy* of + those objects rather than the original ones, and changes made to the copy are + not reflected in the original object. Keep this in mind if you rely on + sharing mutable state through ``cb_kwargs`` or ``meta``. + .. _job-dir-contents: Job directory contents diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst index a0fff6acb..67b435067 100644 --- a/docs/topics/request-response.rst +++ b/docs/topics/request-response.rst @@ -188,6 +188,13 @@ Request objects ``failure.request.cb_kwargs`` in the request's errback. For more information, see :ref:`errback-cb_kwargs`. + .. note:: When :setting:`JOBDIR` is set, requests are serialized to disk + with :mod:`pickle` (see :ref:`request-serialization`). As a result, + the callback receives a deep copy of any object stored in + ``cb_kwargs``, so mutating such an object in the callback does not + affect the original. Avoid relying on shared mutable state passed + through ``cb_kwargs`` in that case. + .. attribute:: Request.meta :value: {}