added tips about using Firefox addons to inspect the live browser DOM

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401051

scrapy/trunk/docs/topics/firebug.rst

@@ -9,7 +9,8 @@ Introduction
 
 This document explains how to use `Firebug`_ (a Firefox add-on) to make the
 scraping process easier and more fun. For other useful Firefox add-ons see
-:ref:`topics-firefox`.
+:ref:`topics-firefox-addons`. There are some caveats with using Firefox add-ons
+to inspect pages, see :ref:`topics-firefox-livedom`.
 
 In this example, we'll show how to use `Firebug`_ to scrape data from the
 `Google Directory`_, which contains the same data as the `Open Directory
@@ -152,3 +153,10 @@ Finally, we can write our ``parse_category()`` method::
             item.description = link.x('font[2]/text()').extract()
             yield item
 
+
+make sure you you may find some elements which appear in Firebug but
+doesn't belong to the original HTML, such as the typical case of ``<tbody>``
+elements.
+
+or tags which Therefer   in page HTML
+sources may on Firebug inspects the live DOM 

scrapy/trunk/docs/topics/firefox.rst

@@ -1,14 +1,44 @@
 .. _topics-firefox:
 
-===================================
+==========================
+Using Firefox for scraping
+==========================
+
+Here is a list of tips and advices on using Firefox for scraping, along with a
+list of useful Firefox add-ons to ease the scraping process.
+
+.. _topics-firefox-livedom:
+
+Caveats with inspecting the live browser DOM
+============================================
+
+Since Firefox add-ons operate on a live browser DOM, what you'll actually see
+when inspecting the page source is not the original HTML, but a modified one
+after applying some browser clean up and executing Javascript code.  Firefox,
+in particular, is known for adding ``<tbody>`` elements to tables.  Scrapy, on
+the other hand, does not modify the original page HTML, so you won't be able to
+extract any data if you use ``<tbody`` in your XPath expressions. 
+
+Therefore, you should keep in mind the following things when working with
+Firefox and XPath:
+
+* Disable Firefox Javascript while inspecting the DOM looking for XPaths to be
+  used in Scrapy
+
+* Never use full XPath paths, use relative and clever ones based on attributes
+  (such as ``id``, ``class``, ``width``, etc) or any identifying features like
+  ``contains(@href, 'image')``.
+
+* Never include ``<tbody>`` elements in your XPath expressions unless you
+  really know what you're doing
+
+.. _topics-firefox-addons:
+
 Useful Firefox add-ons for scraping
 ===================================
 
-Here is a list of useful Firefox add-ons you can use to make the scraping
-process easier, faster and more comfortable.
-
 Firebug
-=======
+-------
 
 `Firebug`_ is a widely known tool among web developers and it's also very
 useful for scraping. In particular, its `Inspect Element`_ feature comes very
@@ -20,24 +50,24 @@ See :ref:`topics-firebug` for a detailed guide on how to use Firebug with
 Scrapy.
 
 XPather
-=======
+-------
 
 `XPather`_ allows you to test XPath expressions directly on the pages.
 
 XPath Checker
-=============
+-------------
 
 `XPath Checker`_ is another Firefox add-on for testing XPaths on your pages.
 
 Tamper Data
-===========
+-----------
 
 `Tamper Data`_ is a Firefox add-on which allows you to view and modify the HTTP
 request headers sent by Firefox. Firebug also allows to view HTTP headers, but
 not to modify them.
 
 Firecookie
-==========
+----------
 
 `Firecookie`_ makes it easier to view and manage cookies. You can use this
 extension to create a new cookie, delete existing cookies, see list of cookies

scrapy/trunk/docs/topics/index.rst

@@ -22,5 +22,5 @@ This section describes all key concepts of Scrapy.
    stats
    webconsole
    robotstxt
-   firebug
    firefox
+   firebug