From bdbca1e2d7b4bf13332eec24968d8a7b9aea2de4 Mon Sep 17 00:00:00 2001 From: Mikhail Korobov Date: Sun, 21 Sep 2014 07:30:44 +0600 Subject: [PATCH] DOC request queue memory usage --- docs/topics/leaks.rst | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/docs/topics/leaks.rst b/docs/topics/leaks.rst index 95bb882e9..d6530859b 100644 --- a/docs/topics/leaks.rst +++ b/docs/topics/leaks.rst @@ -43,12 +43,22 @@ properly. For example, allocating resources on :signal:`spider_opened` but not releasing them on :signal:`spider_closed` may cause problems if you're running :ref:`multiple spiders per process `. +Too Many Requests? +------------------ + +By default Scrapy keeps the request queue in memory; it includes +:class:`~scrapy.http.Request` objects and all objects +referenced in Request attributes (e.g. in :attr:`~scrapy.http.Request.meta`). +While not necesserily a leak, this can take a lot of memory. Enabling +:ref:`persistent job queue ` could help keeping memory usage +in control. + .. _topics-leaks-trackrefs: Debugging memory leaks with ``trackref`` ======================================== -``trackref`` is a module provided by Scrapy to debug the most common cases of +:mod:`trackref` is a module provided by Scrapy to debug the most common cases of memory leaks. It basically tracks the references to all live Requests, Responses, Item and Selector objects.