diff --git a/scrapy/trunk/docs/topics/firebug.rst b/scrapy/trunk/docs/topics/firebug.rst index c62f9530f..6c57f7913 100644 --- a/scrapy/trunk/docs/topics/firebug.rst +++ b/scrapy/trunk/docs/topics/firebug.rst @@ -9,7 +9,8 @@ Introduction This document explains how to use `Firebug`_ (a Firefox add-on) to make the scraping process easier and more fun. For other useful Firefox add-ons see -:ref:`topics-firefox`. +:ref:`topics-firefox-addons`. There are some caveats with using Firefox add-ons +to inspect pages, see :ref:`topics-firefox-livedom`. In this example, we'll show how to use `Firebug`_ to scrape data from the `Google Directory`_, which contains the same data as the `Open Directory @@ -152,3 +153,10 @@ Finally, we can write our ``parse_category()`` method:: item.description = link.x('font[2]/text()').extract() yield item + +make sure you you may find some elements which appear in Firebug but +doesn't belong to the original HTML, such as the typical case of ``
`` +elements. + +or tags which Therefer in page HTML +sources may on Firebug inspects the live DOM diff --git a/scrapy/trunk/docs/topics/firefox.rst b/scrapy/trunk/docs/topics/firefox.rst index 0d1a06f4e..f06633857 100644 --- a/scrapy/trunk/docs/topics/firefox.rst +++ b/scrapy/trunk/docs/topics/firefox.rst @@ -1,14 +1,44 @@ .. _topics-firefox: -=================================== +========================== +Using Firefox for scraping +========================== + +Here is a list of tips and advices on using Firefox for scraping, along with a +list of useful Firefox add-ons to ease the scraping process. + +.. _topics-firefox-livedom: + +Caveats with inspecting the live browser DOM +============================================ + +Since Firefox add-ons operate on a live browser DOM, what you'll actually see +when inspecting the page source is not the original HTML, but a modified one +after applying some browser clean up and executing Javascript code. Firefox, +in particular, is known for adding ```` elements to tables. Scrapy, on +the other hand, does not modify the original page HTML, so you won't be able to +extract any data if you use ```` elements in your XPath expressions unless you + really know what you're doing + +.. _topics-firefox-addons: + Useful Firefox add-ons for scraping =================================== -Here is a list of useful Firefox add-ons you can use to make the scraping -process easier, faster and more comfortable. - Firebug -======= +------- `Firebug`_ is a widely known tool among web developers and it's also very useful for scraping. In particular, its `Inspect Element`_ feature comes very @@ -20,24 +50,24 @@ See :ref:`topics-firebug` for a detailed guide on how to use Firebug with Scrapy. XPather -======= +------- `XPather`_ allows you to test XPath expressions directly on the pages. XPath Checker -============= +------------- `XPath Checker`_ is another Firefox add-on for testing XPaths on your pages. Tamper Data -=========== +----------- `Tamper Data`_ is a Firefox add-on which allows you to view and modify the HTTP request headers sent by Firefox. Firebug also allows to view HTTP headers, but not to modify them. Firecookie -========== +---------- `Firecookie`_ makes it easier to view and manage cookies. You can use this extension to create a new cookie, delete existing cookies, see list of cookies diff --git a/scrapy/trunk/docs/topics/index.rst b/scrapy/trunk/docs/topics/index.rst index abe9b18be..b60b0826b 100644 --- a/scrapy/trunk/docs/topics/index.rst +++ b/scrapy/trunk/docs/topics/index.rst @@ -22,5 +22,5 @@ This section describes all key concepts of Scrapy. stats webconsole robotstxt - firebug firefox + firebug