Document cb_kwargs/meta deep-copy when JOBDIR is set (#7573)

When JOBDIR is enabled, requests are serialized to disk with pickle, so
the objects stored in a request's cb_kwargs and meta are deep-copied on
the round trip. Callbacks then receive copies rather than the original
objects, which is easy to miss and can silently break code that relies on
sharing mutable state. Add a note to the request serialization section of
the jobs docs and a cross-referenced caution to the Request.cb_kwargs
attribute docs.

Closes #6120
This commit is contained in:
Labib Bin Salam 2026-06-11 10:40:15 +01:00 committed by GitHub
parent e7f229f5b2
commit e0a7de7213
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
2 changed files with 16 additions and 0 deletions

View File

@ -104,6 +104,15 @@ If you wish to log the requests that couldn't be serialized, you can set the
:setting:`SCHEDULER_DEBUG` setting to ``True`` in the project's settings page.
It is ``False`` by default.
.. note:: Because requests are serialized with :mod:`pickle`, the objects you
store on a request, such as the values of its
:attr:`~scrapy.Request.cb_kwargs` and :attr:`~scrapy.Request.meta`
dictionaries, are deep-copied when the request is written to and later read
back from the job directory. As a result, the callback receives a *copy* of
those objects rather than the original ones, and changes made to the copy are
not reflected in the original object. Keep this in mind if you rely on
sharing mutable state through ``cb_kwargs`` or ``meta``.
.. _job-dir-contents:
Job directory contents

View File

@ -188,6 +188,13 @@ Request objects
``failure.request.cb_kwargs`` in the request's errback. For more information,
see :ref:`errback-cb_kwargs`.
.. note:: When :setting:`JOBDIR` is set, requests are serialized to disk
with :mod:`pickle` (see :ref:`request-serialization`). As a result,
the callback receives a deep copy of any object stored in
``cb_kwargs``, so mutating such an object in the callback does not
affect the original. Avoid relying on shared mutable state passed
through ``cb_kwargs`` in that case.
.. attribute:: Request.meta
:value: {}