diff --git a/docs/topics/jobs.rst b/docs/topics/jobs.rst index 769925dd5..8d6b2458b 100644 --- a/docs/topics/jobs.rst +++ b/docs/topics/jobs.rst @@ -104,6 +104,15 @@ If you wish to log the requests that couldn't be serialized, you can set the :setting:`SCHEDULER_DEBUG` setting to ``True`` in the project's settings page. It is ``False`` by default. +.. note:: Because requests are serialized with :mod:`pickle`, the objects you + store on a request, such as the values of its + :attr:`~scrapy.Request.cb_kwargs` and :attr:`~scrapy.Request.meta` + dictionaries, are deep-copied when the request is written to and later read + back from the job directory. As a result, the callback receives a *copy* of + those objects rather than the original ones, and changes made to the copy are + not reflected in the original object. Keep this in mind if you rely on + sharing mutable state through ``cb_kwargs`` or ``meta``. + .. _job-dir-contents: Job directory contents diff --git a/docs/topics/request-response.rst b/docs/topics/request-response.rst index a0fff6acb..67b435067 100644 --- a/docs/topics/request-response.rst +++ b/docs/topics/request-response.rst @@ -188,6 +188,13 @@ Request objects ``failure.request.cb_kwargs`` in the request's errback. For more information, see :ref:`errback-cb_kwargs`. + .. note:: When :setting:`JOBDIR` is set, requests are serialized to disk + with :mod:`pickle` (see :ref:`request-serialization`). As a result, + the callback receives a deep copy of any object stored in + ``cb_kwargs``, so mutating such an object in the callback does not + affect the original. Avoid relying on shared mutable state passed + through ``cb_kwargs`` in that case. + .. attribute:: Request.meta :value: {}