Merge pull request #149 from alexcepoi/parse-changes

documentation for `parse` command, debugging spiders section

docs/index.rst

@@ -128,6 +128,7 @@ Solving specific problems
    :hidden:
 
    faq
+   topics/debug
    topics/firefox
    topics/firebug
    topics/leaks
@@ -139,6 +140,9 @@ Solving specific problems
 :doc:`faq`
     Get answers to most frequently asked questions.
 
+:doc:`topics/debug`
+    Learn how to debug common problems of your scrapy spider.
+
 :doc:`topics/firefox`
     Learn how to scrape with Firefox and some useful add-ons.
 

docs/topics/commands.rst

@@ -368,15 +368,25 @@ Supported options:
 
 * ``--nolinks``: don't show scraped items
 
+* ``--depth`` or ``-d``: depth level for which the requests should be followed
+  recursively (default: 1)
+
+* ``--verbose`` or ``-v``: display information for each depth level
+
 Usage example::
 
     $ scrapy parse http://www.example.com/ -c parse_item
     [ ... scrapy log lines crawling example.com spider ... ]
-    # Scraped Items - callback: parse ------------------------------------------------------------
-    MyItem({'name': u"Example item",
+
+    >>> STATUS DEPTH LEVEL 1 <<<
+    # Scraped Items  ------------------------------------------------------------
+    [{'name': u'Example item',
      'category': u'Furniture',
-     'length': u'12 cm'}
-    )
+     'length': u'12 cm'}]
+
+    # Requests  -----------------------------------------------------------------
+    []
+
 
 .. command:: settings
 

docs/topics/debug.rst

@@ -0,0 +1,125 @@
+.. _topics-debug:
+
+=================
+Debugging Spiders
+=================
+
+This document explains the most common techniques for debugging spiders.
+Consider the following scrapy spider below::
+
+    class MySpider(BaseSpider):
+        name = 'myspider'
+        start_urls = (
+            'http://example.com/page1',
+            'http://example.com/page2',
+            )
+
+        def parse(self, response):
+            # collect `item_urls`
+            for item_url in item_urls:
+                yield Request(url=item_url, callback=self.parse_item)
+
+        def parse_item(self, response):
+            item = MyItem()
+            # populate `item` fields
+            yield Request(url=item_details_url, meta={'item': item},
+                callback=self.parse_details)
+
+        def parse_details(self, response):
+            item = response.meta['item']
+            # populate more `item` fields
+            return item
+
+Basically this is a simple spider which parses two pages of items (the
+start_urls). Items also have a details page with additional information, so we
+use the ``meta`` functionality of :class:`~scrapy.http.Request` to pass a
+partially populated item.
+
+
+Parse Command
+=============
+
+The most basic way of checking the output of your spider is to use the
+:command:`parse` command. It allows to check the behaviour of different parts
+of the spider at the method level. It has the advantage of being flexible and
+simple to use, but does not allow debugging code inside a method.
+
+In order to see the item scraped from a specific url::
+
+    $ scrapy --spider=myspider -c parse_item -d 2 <item_url>
+    [ ... scrapy log lines crawling example.com spider ... ]
+
+    >>> STATUS DEPTH LEVEL 2 <<<
+    # Scraped Items  ------------------------------------------------------------
+    [{'url': <item_url>}]
+
+    # Requests  -----------------------------------------------------------------
+    []
+
+Using the ``--verbose`` or ``-v`` option we can see the status at each depth level::
+
+    $ scrapy --spider=myspider -c parse_item -d 2 -v <item_url>
+    [ ... scrapy log lines crawling example.com spider ... ]
+
+    >>> DEPTH LEVEL: 1 <<<
+    # Scraped Items  ------------------------------------------------------------
+    []
+
+    # Requests  -----------------------------------------------------------------
+    [<GET item_details_url>]
+
+
+    >>> DEPTH LEVEL: 2 <<<
+    # Scraped Items  ------------------------------------------------------------
+    [{'url': <item_url>}]
+
+    # Requests  -----------------------------------------------------------------
+    []
+
+Checking items scraped from a single start_url, can also be easily achieved
+using::
+
+    $ scrapy --spider=myspider -d 3 'http://example.com/page1'
+
+
+Scrapy Shell
+============
+
+While the :command:`parse` command is very useful for checking behaviour of a
+spider, it is of little help to check what happens inside a callback, besides
+showing the reponse received and the output. How to debug the situation when
+``parse_details`` sometimes receives no item?
+
+Fortunately, the :command:`shell` is your bread and butter in this case (see
+:ref:`topics-shell-inspect-response`)::
+
+    from scrapy.shell import inspect_response
+
+    def parse_details(self, response):
+        item = response.meta.get('item', None)
+        if item:
+            # populate more `item` fields
+            return item
+        else:
+            inspect_response(response, self)
+
+
+Logging
+=======
+
+Logging is another useful option for getting information about your spider run.
+Although not as convenient, it comes with the advantage that the logs will be
+available in all future runs should they be necessary again::
+
+    from scrapy import log
+
+    def parse_details(self, response):
+        item = response.meta.get('item', None)
+        if item:
+            # populate more `item` fields
+            return item
+        else:
+            self.log('No item received for %s' % response.url,
+                level=log.WARNING)
+
+For more information, check the :ref:`topics-logging` section.

scrapy/commands/parse.py

@@ -154,7 +154,7 @@ class Command(ScrapyCommand):
 
             cb = cb if callable(cb) else getattr(self.spider, cb, None)
             if not cb:
-                log.msg('Cannot find callback %r in spider: %s' % (callback, spider.name))
+                log.msg('Cannot find callback %r in spider: %s' % (callback, self.spider.name))
 
             # parse items and requests
             depth = response.meta['_depth']