From 46aaea9f32b38a7383a8829eab1960cc5d9869ca Mon Sep 17 00:00:00 2001 From: Pablo Hoffman Date: Sun, 23 Aug 2009 20:36:00 -0300 Subject: [PATCH] doc: improved documentation about debugging leaks --- docs/experimental/exporters.rst | 2 +- docs/topics/leaks.rst | 63 ++++++++++++++++++++++++++++----- 2 files changed, 55 insertions(+), 10 deletions(-) diff --git a/docs/experimental/exporters.rst b/docs/experimental/exporters.rst index d03ec404c..51c65877e 100644 --- a/docs/experimental/exporters.rst +++ b/docs/experimental/exporters.rst @@ -267,7 +267,7 @@ PickleItemExporter This isn't a human readable format, so no output examples are provided. -.. _pickle module: http://docs.python.org/library/pickle.html +.. _pickle module documentation: http://docs.python.org/library/pickle.html PprintItemExporter ------------------ diff --git a/docs/topics/leaks.rst b/docs/topics/leaks.rst index 835c8cb4c..9323595b5 100644 --- a/docs/topics/leaks.rst +++ b/docs/topics/leaks.rst @@ -79,19 +79,63 @@ class. If you do have leaks, chances are you can figure out which spider is leaking by looking at the oldest request or response. You can get the oldest object of -each class using the :func:`get_oldest` function like this (from the telnet -console):: +each class using the :func:`~scrapy.utils.trackref.get_oldest` function like +this (from the telnet console). + +A real example +-------------- + +Let's see a concrete example of an hypothetical case of memory leaks. + +Suppose we have some spider with a line similar to this one:: + + return Request("http://www.somenastyspider.com/product.php?pid=%d" % product_id, + callback=self.parse, meta={referer: response}") + +That line is passing a response reference inside a request which effectively +ties the response lifetime to the requests one, and that's would definitely +cause memory leaks. + +Let's see how we can discover which one is the nasty spider (without knowing it +a-priori, of course) by using the ``trackref`` tool. + +After the crawler is running for a few minutes and we notice its memory usage +has grown a lot, we can enter its telnet console and check the live +references:: + + >>> prefs() + Live References + + HtmlResponse 3890 oldest: 265s ago + XPathSelector 2 oldest: 0s ago + Request 3878 oldest: 250s ago + +The fact that there are so many live responses (and that they're so old) is +definitely suspicious, as responses should have a relatively short lifetime +compared to Requests. So let's check the oldest response:: >>> from scrapy.utils.trackref import get_oldest - >>> req = get_oldest('FormRequest') - >>> req.url - 'http://www.example.com/ecommerce/product.php?pid=123' + >>> r = get_oldest('HtmlResponse') + >>> r.url + 'http://www.somenastyspider.com/product.php?pid=123' + +There it is. By looking at the URL of the oldest response we can see it belongs +to the ``somenastyspider.com`` spider. We can now go and check the code of that +spider to discover the nasty line that is generating the leaks (passing +response references inside requests). + +.. module:: scrapy.utils.trackref + :synopsis: Track references of live objects scrapy.utils.trackref module ---------------------------- -.. module:: scrapy.utils.trackref - :synopsis: Track references of live objects +Here are the functions available in the :mod:`~scrapy.utils.trackref` module. + +.. class:: object_ref + + Inherit from this class (instead of object) if you want to track live + instances with the ``trackref`` module. .. function:: print_live_refs(class_name) @@ -99,8 +143,9 @@ scrapy.utils.trackref module .. function:: get_oldest(class_name) - Return the old object alive from the given class name, or ``None`` if not - found. + Return the oldest object alive with the given class name, or ``None`` if + none is found. Use :func:`print_live_refs` first to get a list of all + tracked live objects. .. _topics-leaks-guppy: