12 KiB
Debugging memory leaks
In Scrapy, objects such as requests, responses and items have a finite lifetime: they are created, used for a while and finally destroyed.
Of all those objects, the Request usually has the longest lifetime because it waits in the Scheduler queue until it's time to process it. For more information, see :ref:`topics-architecture`.
System Message: ERROR/3 (<stdin>, line 10); backlink
Unknown interpreted text role "ref".Because these Scrapy objects have a comparatively long lifetime, there is always the risk of accumulating them in memory without releasing them properly and thus causing what is known as a "memory leak".
To help debug memory leaks, Scrapy provides a built-in mechanism for tracking object references called :ref:`trackref <topics-leaks-trackrefs>`, and you can also use a third-party library called :ref:`muppy <topics-leaks-muppy>` for more advanced memory debugging (see below for more information). Both mechanisms must be used from the :ref:`Telnet Console <topics-telnetconsole>`.
System Message: ERROR/3 (<stdin>, line 18); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 18); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 18); backlink
Unknown interpreted text role "ref".Common causes of memory leaks
It happens quite often (sometimes by accident, sometimes on purpose) that the Scrapy developer passes objects referenced in requests—for example, through the :attr:`~scrapy.Request.cb_kwargs` or :attr:`~scrapy.Request.meta` attributes or the request callback function—and that effectively ties the lifetime of those referenced objects to the lifetime of the request. This is, by far, the most common cause of memory leaks in Scrapy projects, and it can be quite difficult for newcomers to debug.
System Message: ERROR/3 (<stdin>, line 27); backlink
Unknown interpreted text role "attr".System Message: ERROR/3 (<stdin>, line 27); backlink
Unknown interpreted text role "attr".In big projects, spiders are typically written by different people, and some of those spiders could be "leaking" and thus affecting the rest of the (well-written) spiders when they run concurrently, which, in turn, affects the whole crawling process.
The leak could also come from a custom middleware, pipeline, or extension that you wrote if you are not releasing the previously allocated resources properly. For example, allocating resources on :signal:`spider_opened` but not releasing them on :signal:`spider_closed` may cause problems if you're running :ref:`multiple spiders per process <run-multiple-spiders>`.
System Message: ERROR/3 (<stdin>, line 40); backlink
Unknown interpreted text role "signal".System Message: ERROR/3 (<stdin>, line 40); backlink
Unknown interpreted text role "signal".System Message: ERROR/3 (<stdin>, line 40); backlink
Unknown interpreted text role "ref".Too Many Requests?
By default Scrapy keeps the request queue in memory; it includes :class:`~scrapy.Request` objects and all objects referenced in request attributes (for example, :attr:`~scrapy.Request.cb_kwargs` and :attr:`~scrapy.Request.meta`). While this is not necessarily a leak, it can consume a lot of memory. Enabling the :ref:`persistent job queue <topics-jobs>` can help keep memory usage under control.
System Message: ERROR/3 (<stdin>, line 49); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 49); backlink
Unknown interpreted text role "attr".System Message: ERROR/3 (<stdin>, line 49); backlink
Unknown interpreted text role "attr".System Message: ERROR/3 (<stdin>, line 49); backlink
Unknown interpreted text role "ref".Debugging memory leaks with trackref
:mod:`trackref` is a module provided by Scrapy to debug the most common cases of memory leaks. It tracks the references to all live Request, Response, Item, Spider and Selector objects.
System Message: ERROR/3 (<stdin>, line 63); backlink
Unknown interpreted text role "mod".You can enter the telnet console and inspect how many objects (of the classes mentioned above) are currently alive using the prefs() function which is an alias to the :func:`~scrapy.utils.trackref.print_live_refs` function:
System Message: ERROR/3 (<stdin>, line 67); backlink
Unknown interpreted text role "func".telnet localhost 6023
.. code-block:: pycon
>>> prefs()
Live References
ExampleSpider 1 oldest: 15s ago
HtmlResponse 10 oldest: 1s ago
Selector 2 oldest: 0s ago
FormRequest 878 oldest: 7s ago
As you can see, that report also shows the "age" of the oldest object in each class. If you're running multiple spiders per process, chances are you can figure out which spider is leaking by looking at the oldest request or response. You can get the oldest object of each class using the :func:`~scrapy.utils.trackref.get_oldest` function (from the telnet console).
System Message: ERROR/3 (<stdin>, line 83); backlink
Unknown interpreted text role "func".Which objects are tracked?
trackref tracks objects from these classes (and all their subclasses):
-
System Message: ERROR/3 (<stdin>, line 94); backlink
Unknown interpreted text role "class".
-
System Message: ERROR/3 (<stdin>, line 95); backlink
Unknown interpreted text role "class".
-
System Message: ERROR/3 (<stdin>, line 96); backlink
Unknown interpreted text role "class".
-
System Message: ERROR/3 (<stdin>, line 97); backlink
Unknown interpreted text role "class".
-
System Message: ERROR/3 (<stdin>, line 98); backlink
Unknown interpreted text role "class".
A real example
Let's see a concrete example of a hypothetical case of memory leaks. Suppose we have some spider with a line similar to this one:
return Request(f"http://www.somenastyspider.com/product.php?pid={product_id}",
callback=self.parse, cb_kwargs={'referer': response})
That line passes a response reference inside a request, which effectively ties the response lifetime to the request's lifetime, and that will definitely cause memory leaks.
Let's see how we can discover the cause (without knowing it a priori, of course) by using the trackref tool.
After the crawler is running for a few minutes and we notice its memory usage has grown a lot, we can enter its telnet console and check the live references:
System Message: WARNING/2 (<stdin>, line 120)
Cannot analyze code. Pygments package not found.
.. code-block:: pycon
>>> prefs()
Live References
SomenastySpider 1 oldest: 15s ago
HtmlResponse 3890 oldest: 265s ago
Selector 2 oldest: 0s ago
Request 3878 oldest: 250s ago
The fact that there are so many live responses (and that they're so old) is definitely suspicious, as responses should have a relatively short lifetime compared to requests. The number of responses is similar to the number of requests, so it looks like they are tied in some way. We can now check the code of the spider to discover the line that is generating the leaks (passing response references inside requests).
Sometimes extra information about live objects can be helpful. Let's check the oldest response:
System Message: WARNING/2 (<stdin>, line 140)
Cannot analyze code. Pygments package not found.
.. code-block:: pycon
>>> from scrapy.utils.trackref import get_oldest
>>> r = get_oldest("HtmlResponse")
>>> r.url
'http://www.somenastyspider.com/product.php?pid=123'
If you want to iterate over all objects instead of getting only the oldest one, you can use the :func:`scrapy.utils.trackref.iter_all` function:
System Message: ERROR/3 (<stdin>, line 147); backlink
Unknown interpreted text role "func".System Message: WARNING/2 (<stdin>, line 150)
Cannot analyze code. Pygments package not found.
.. code-block:: pycon
>>> from scrapy.utils.trackref import iter_all
>>> [r.url for r in iter_all("HtmlResponse")]
['http://www.somenastyspider.com/product.php?pid=123',
'http://www.somenastyspider.com/product.php?pid=584',
...]
Too many spiders?
If your project has too many spiders executed in parallel, the output of :func:`prefs` can be difficult to read. For this reason, that function has an ignore argument that you can use to omit a particular class (and all its subclasses). For example, this won't show any live references to spiders:
System Message: ERROR/3 (<stdin>, line 161); backlink
Unknown interpreted text role "func".System Message: WARNING/2 (<stdin>, line 166)
Cannot analyze code. Pygments package not found.
.. code-block:: pycon
>>> from scrapy.spiders import Spider
>>> prefs(ignore=Spider)
System Message: ERROR/3 (<stdin>, line 171)
Unknown directive type "module".
.. module:: scrapy.utils.trackref :synopsis: Track references of live objects
scrapy.utils.trackref module
Here are the functions available in the :mod:`~scrapy.utils.trackref` module.
System Message: ERROR/3 (<stdin>, line 177); backlink
Unknown interpreted text role "mod".Inherit from this class if you want to track live instances with the trackref module.
System Message: ERROR/3 (<stdin>, line 184)
Unknown directive type "function".
.. function:: print_live_refs(class_name, ignore=NoneType)
Print a report of live references, grouped by class name.
:param ignore: if given, all objects from the specified class (or tuple of
classes) will be ignored.
:type ignore: type or tuple
System Message: ERROR/3 (<stdin>, line 192)
Unknown directive type "function".
.. function:: get_oldest(class_name)
Return the oldest object alive with the given class name, or ``None`` if
none is found. Use :func:`print_live_refs` first to get a list of all
tracked live objects per class name.
System Message: ERROR/3 (<stdin>, line 198)
Unknown directive type "function".
.. function:: iter_all(class_name)
Return an iterator over all objects alive with the given class name, or
``None`` if none is found. Use :func:`print_live_refs` first to get a list
of all tracked live objects per class name.
Debugging memory leaks with muppy
trackref provides a convenient mechanism for tracking down memory leaks, but it only keeps track of the objects that are more likely to cause them. However, sometimes leaks come from other (more or less obscure) objects. If that happens and you can't find your leaks using trackref, you still have another resource: the muppy library.
muppy is available as part of Pympler.
If you use pip, you can install muppy with the following command:
pip install Pympler
Here's an example that shows all Python objects available in the heap using muppy:
System Message: WARNING/2 (<stdin>, line 229)
Cannot analyze code. Pygments package not found.
.. code-block:: pycon
>>> from pympler import muppy
>>> all_objects = muppy.get_objects()
>>> len(all_objects)
28667
>>> from pympler import summary
>>> suml = summary.summarize(all_objects)
>>> summary.print_(suml)
types | # objects | total size
==================================== | =========== | ============
<class 'str | 9822 | 1.10 MB
<class 'dict | 1658 | 856.62 KB
<class 'type | 436 | 443.60 KB
<class 'code | 2974 | 419.56 KB
<class '_io.BufferedWriter | 2 | 256.34 KB
<class 'set | 420 | 159.88 KB
<class '_io.BufferedReader | 1 | 128.17 KB
<class 'wrapper_descriptor | 1130 | 88.28 KB
<class 'tuple | 1304 | 86.57 KB
<class 'weakref | 1013 | 79.14 KB
<class 'builtin_function_or_method | 958 | 67.36 KB
<class 'method_descriptor | 865 | 60.82 KB
<class 'abc.ABCMeta | 62 | 59.96 KB
<class 'list | 446 | 58.52 KB
<class 'int | 1425 | 43.20 KB
For more information about muppy, refer to the muppy documentation.
Leaks without leaks
Sometimes you may notice that the memory usage of your Scrapy process only increases and never decreases. Unfortunately, this could happen even though neither Scrapy nor your project are leaking memory. This is due to a not-so-well-known problem in Python, which may not return released memory to the operating system in some cases. For more information on this issue see:
The improvements proposed by Evan Jones, which are detailed in this paper, got merged in Python 2.5, but this only reduces the problem, it doesn't fix it completely. To quote the paper:
Unfortunately, this patch can only free an arena if there are no more objects allocated in it anymore. This means that fragmentation is a large issue. An application could have many megabytes of free memory, scattered throughout all the arenas, but it will be unable to free any of it. This is a problem experienced by all memory allocators. The only way to solve it is to move to a compacting garbage collector, which is able to move objects in memory. This would require significant changes to the Python interpreter.
To keep memory consumption reasonable, you can split the job into several smaller jobs or enable the :ref:`persistent job queue <topics-jobs>` and stop and start the spider from time to time.
System Message: ERROR/3 (<stdin>, line 291); backlink
Unknown interpreted text role "ref".