mirror of https://github.com/scrapy/scrapy.git
Document cb_kwargs/meta deep-copy when JOBDIR is set (#7573)
When JOBDIR is enabled, requests are serialized to disk with pickle, so the objects stored in a request's cb_kwargs and meta are deep-copied on the round trip. Callbacks then receive copies rather than the original objects, which is easy to miss and can silently break code that relies on sharing mutable state. Add a note to the request serialization section of the jobs docs and a cross-referenced caution to the Request.cb_kwargs attribute docs. Closes #6120
This commit is contained in:
parent
e7f229f5b2
commit
e0a7de7213
|
|
@ -104,6 +104,15 @@ If you wish to log the requests that couldn't be serialized, you can set the
|
|||
:setting:`SCHEDULER_DEBUG` setting to ``True`` in the project's settings page.
|
||||
It is ``False`` by default.
|
||||
|
||||
.. note:: Because requests are serialized with :mod:`pickle`, the objects you
|
||||
store on a request, such as the values of its
|
||||
:attr:`~scrapy.Request.cb_kwargs` and :attr:`~scrapy.Request.meta`
|
||||
dictionaries, are deep-copied when the request is written to and later read
|
||||
back from the job directory. As a result, the callback receives a *copy* of
|
||||
those objects rather than the original ones, and changes made to the copy are
|
||||
not reflected in the original object. Keep this in mind if you rely on
|
||||
sharing mutable state through ``cb_kwargs`` or ``meta``.
|
||||
|
||||
.. _job-dir-contents:
|
||||
|
||||
Job directory contents
|
||||
|
|
|
|||
|
|
@ -188,6 +188,13 @@ Request objects
|
|||
``failure.request.cb_kwargs`` in the request's errback. For more information,
|
||||
see :ref:`errback-cb_kwargs`.
|
||||
|
||||
.. note:: When :setting:`JOBDIR` is set, requests are serialized to disk
|
||||
with :mod:`pickle` (see :ref:`request-serialization`). As a result,
|
||||
the callback receives a deep copy of any object stored in
|
||||
``cb_kwargs``, so mutating such an object in the callback does not
|
||||
affect the original. Avoid relying on shared mutable state passed
|
||||
through ``cb_kwargs`` in that case.
|
||||
|
||||
.. attribute:: Request.meta
|
||||
:value: {}
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue