From 16d9a339511b7f7f27b41e93bda1ae4e2795700f Mon Sep 17 00:00:00 2001 From: Pablo Hoffman Date: Tue, 15 Feb 2011 07:24:52 -0200 Subject: [PATCH] added FAQ entry about working with big data feeds --- docs/faq.rst | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs/faq.rst b/docs/faq.rst index b401223f7..2444a3c05 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -228,3 +228,15 @@ which scrapes one of these sites. .. _this page: http://search.cpan.org/~ecarroll/HTML-TreeBuilderX-ASP_NET-0.09/lib/HTML/TreeBuilderX/ASP_NET.pm .. _example spider: http://github.com/AmbientLighter/rpn-fas/blob/master/fas/spiders/rnp.py + +What's the best way to parse big XML/CSV data feeds? +---------------------------------------------------- + +Parsing big feeds with XPath selectors can be problematic since they need to +build the DOM of the entire feed in memory, and this can be quite slow and +consume a lot of memory. + +In order to avoid parsing all the entire feed at once in memory, you can use +the functions ``xmliter`` and ``csviter`` from ``scrapy.utils.iterators`` +module. In fact, this is what the feed spiders (see :ref:`topics-spiders`) use +under the cover.