From 7c66f1739dc22494e0d3d272e46eeaa1845f0350 Mon Sep 17 00:00:00 2001 From: Pablo Hoffman Date: Fri, 10 Apr 2009 05:35:53 +0000 Subject: [PATCH] Several documentation changes: - merged (and updated) new tutorial from proposed doc - striped old tutorial and created new firebug topic - added topic about useful third firefox add-ons - rearranged main documentation index - several assorted documentation fixes --HG-- rename : scrapy/trunk/docs/proposed/tutorial.rst => scrapy/trunk/docs/intro/tutorial.rst rename : scrapy/trunk/docs/intro/tutorial/scrot1.png => scrapy/trunk/docs/topics/_images/firebug1.png rename : scrapy/trunk/docs/intro/tutorial/scrot2.png => scrapy/trunk/docs/topics/_images/firebug2.png rename : scrapy/trunk/docs/intro/tutorial/scrot3.png => scrapy/trunk/docs/topics/_images/firebug3.png extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401038 --- scrapy/trunk/docs/faq.rst | 2 +- scrapy/trunk/docs/index.rst | 62 +++++- scrapy/trunk/docs/intro/index.rst | 2 +- scrapy/trunk/docs/intro/install.rst | 9 +- .../docs/{proposed => intro}/tutorial.rst | 206 ++++++++++-------- scrapy/trunk/docs/intro/tutorial/index.rst | 10 - .../trunk/docs/intro/tutorial/tutorial1.rst | 55 ----- .../trunk/docs/intro/tutorial/tutorial2.rst | 158 -------------- .../trunk/docs/intro/tutorial/tutorial3.rst | 150 ------------- .../trunk/docs/intro/tutorial/tutorial4.rst | 78 ------- scrapy/trunk/docs/misc/api-stability.rst | 33 +-- scrapy/trunk/docs/proposed/index.rst | 4 - scrapy/trunk/docs/ref/index.rst | 4 +- scrapy/trunk/docs/ref/selectors.rst | 6 +- scrapy/trunk/docs/ref/settings.rst | 4 +- .../_images/firebug1.png} | Bin .../_images/firebug2.png} | Bin .../_images/firebug3.png} | Bin scrapy/trunk/docs/topics/adaptors.rst | 6 +- scrapy/trunk/docs/topics/architecture.rst | 6 +- scrapy/trunk/docs/topics/firebug.rst | 154 +++++++++++++ scrapy/trunk/docs/topics/firefox.rst | 52 +++++ scrapy/trunk/docs/topics/index.rst | 5 +- scrapy/trunk/docs/topics/selectors.rst | 18 +- scrapy/trunk/docs/topics/shell.rst | 12 +- 25 files changed, 419 insertions(+), 617 deletions(-) rename scrapy/trunk/docs/{proposed => intro}/tutorial.rst (64%) delete mode 100644 scrapy/trunk/docs/intro/tutorial/index.rst delete mode 100644 scrapy/trunk/docs/intro/tutorial/tutorial1.rst delete mode 100644 scrapy/trunk/docs/intro/tutorial/tutorial2.rst delete mode 100644 scrapy/trunk/docs/intro/tutorial/tutorial3.rst delete mode 100644 scrapy/trunk/docs/intro/tutorial/tutorial4.rst rename scrapy/trunk/docs/{intro/tutorial/scrot1.png => topics/_images/firebug1.png} (100%) rename scrapy/trunk/docs/{intro/tutorial/scrot2.png => topics/_images/firebug2.png} (100%) rename scrapy/trunk/docs/{intro/tutorial/scrot3.png => topics/_images/firebug3.png} (100%) create mode 100644 scrapy/trunk/docs/topics/firebug.rst create mode 100644 scrapy/trunk/docs/topics/firefox.rst diff --git a/scrapy/trunk/docs/faq.rst b/scrapy/trunk/docs/faq.rst index e203daf7c..727bd96ed 100644 --- a/scrapy/trunk/docs/faq.rst +++ b/scrapy/trunk/docs/faq.rst @@ -27,7 +27,7 @@ Does Scrapy work with Python 3.0? --------------------------------- No, and there are no plans to port Scrapy to Python 3.0 yet. At the moment -Scrapy requires Python 2.5 or 2.6. +Scrapy works with Python 2.5 or 2.6. Does Scrapy "stole" X from Django? ---------------------------------- diff --git a/scrapy/trunk/docs/index.rst b/scrapy/trunk/docs/index.rst index 5716fa345..26a0bc9e4 100644 --- a/scrapy/trunk/docs/index.rst +++ b/scrapy/trunk/docs/index.rst @@ -1,29 +1,75 @@ .. _index: +============================== Scrapy |version| documentation -================================ +============================== -Welcome! This is the documentation for Scrapy |version|, last updated on |today|. +Welcome! This is the documentation for Scrapy, a screen scraping framework for +Python. -For more information visit the `Scrapy homepage `_. +For more information about the project visit the `Scrapy homepage +`_. -Documentation Contents: +Getting started +=============== + +This section helps you get familiarized with Scrapy and its basic concepts. .. toctree:: :maxdepth: 2 intro/index - topics/index - ref/index - misc/index +Using Scrapy +============ + +This section explains all key concepts of Scrapy. .. toctree:: - :maxdepth: 1 + :maxdepth: 2 + + topics/index + +API reference +============= + +This section documents the :ref:`Scrapy API `. See also +:ref:`misc-api-stability`. + +.. toctree:: + :maxdepth: 2 + + ref/index + +Frequently asked questions +========================== + +.. toctree:: + :maxdepth: 2 faq +Experimental features +===================== + +This section documents experimental features that may become stable in future +Scrapy releases, but whose API is not yet stable. Use them with caution, and +subscribe to the `mailing lists `_ to be notified +of any changes. + .. toctree:: :maxdepth: 1 + topics/adaptors + +Proposed documentation +====================== + +This section contains documentation which is not yet completed, or which +overlaps with existing documentation and needs to be manually merged. It may +also contain outdated information, as it's not revised so frequently. + +.. toctree:: + :maxdepth: 2 + proposed/index diff --git a/scrapy/trunk/docs/intro/index.rst b/scrapy/trunk/docs/intro/index.rst index 9a64471af..d9fd05d33 100644 --- a/scrapy/trunk/docs/intro/index.rst +++ b/scrapy/trunk/docs/intro/index.rst @@ -12,7 +12,7 @@ might help you. overview install - tutorial/index + tutorial .. seealso:: diff --git a/scrapy/trunk/docs/intro/install.rst b/scrapy/trunk/docs/intro/install.rst index 9fa1081d5..c059ec666 100644 --- a/scrapy/trunk/docs/intro/install.rst +++ b/scrapy/trunk/docs/intro/install.rst @@ -1,8 +1,11 @@ .. _intro-install: -============ -Installation -============ +================== +Installation guide +================== + +This document describes how to install Scrapy in Linux, Windows and Mac OS X +systems. .. highlight:: sh diff --git a/scrapy/trunk/docs/proposed/tutorial.rst b/scrapy/trunk/docs/intro/tutorial.rst similarity index 64% rename from scrapy/trunk/docs/proposed/tutorial.rst rename to scrapy/trunk/docs/intro/tutorial.rst index 0a8a75120..17ce06430 100644 --- a/scrapy/trunk/docs/proposed/tutorial.rst +++ b/scrapy/trunk/docs/intro/tutorial.rst @@ -1,21 +1,23 @@ -.. _tutorial: +.. _intro-tutorial: =============== Scrapy Tutorial =============== -In this tutorial, we'll assume that Scrapy is already installed in your system, -if not see :ref:`intro-install`. +In this tutorial, we'll assume that Scrapy is already installed in your system. +If that's not the case see :ref:`intro-install`. We are going to use `Open directory project (dmoz) `_ as our example domain to scrape. -This tutorial will introduce you to this tasks: +This tutorial will walk you through through these tasks: -* Creating a project -* Defining the Items you will extract -* Writing a spider to crawl a site and extract Items -* Write an Item Pipeline to store the extracted Items +1. Creating a new Scrapy project +2. Defining the Items you will extract +3. Writing a :ref:`spider ` to crawl a site and extract + :ref:`Items ` +4. Writing an :ref:`Item Pipeline ` to store the + extracted Items Creating a project ================== @@ -73,10 +75,11 @@ Our first Spider Spiders are user written classes to scrape information from a domain (or group of domains). -They define an initial set of URLs to download, and how to parse the downloaded contents in the search for data (Items) or more URLs to follow. +They define an initial list of URLs to download, how to follow links, and how +to parse the contents of those pages to extract :ref:`items `. To create a Spider, you must subclass :class:`scrapy.spider.BaseSpider`, and -then define the three main, mandatory, attributes: +define the three main, mandatory, attributes: * :attr:`~scrapy.spider.BaseSpider.domain_name`: identifies the Spider. It must be unique, that is, you can't set the same domain name for different Spiders. @@ -86,16 +89,20 @@ then define the three main, mandatory, attributes: listed here. The subsequent URLs will be generated successively from data contained in the start URLs. -* :meth:`~scrapy.spider.BaseSpider.parse` is the callback method of the spider. - This means that each time a URL is retrieved, the downloaded data (Response) - will be passed to this method. +* :meth:`~scrapy.spider.BaseSpider.parse` is a method of the spider, which will + be called with the downloaded :class:`~scrapy.http.Response` object of each + start URL. The response is passed to the method as the first and only + argument. + This method is responsible for parsing the response data and extracting + scraped data (as scraped items) and more URLs to follow. + The :meth:`~scrapy.spider.BaseSpider.parse` method is in charge of processing - the response and returning scraped data and or more URLs to follow, because of - this, the method must always return a list or at least an empty one. + the response and returning scraped data (as :class:`~scrapy.item.ScrapedItem` + objects) and more URLs to follow (as :class:`~scrapy.http.Request` objects). This is the code for our first Spider, save it in a file named -``dmoz_spider.py`` inside ``dmoz/spiders`` directory:: +``dmoz_spider.py`` under the ``dmoz/spiders`` directory:: from scrapy.spider import BaseSpider @@ -113,19 +120,15 @@ This is the code for our first Spider, save it in a file named SPIDER = DmozSpider() -.. warning:: - - When creating spiders, be sure not to name them equal to the project's name - or you won't be able to import modules from your project in your spider! - Crawling -------- To put our spider to work, go to the project's top level directory and run:: - ./scrapy-ctl.py crawl dmoz.org + python scrapy-ctl.py crawl dmoz.org -The ``crawl dmoz.org`` subcommand runs the spider for the ``dmoz.org`` domain, you'll get an output like this:: +The ``crawl dmoz.org`` subcommand runs the spider for the ``dmoz.org`` domain, +you'll get an output like this:: [-] Log opened. [dmoz] INFO: Enabled extensions: TelnetConsole, WebConsole @@ -145,8 +148,8 @@ The ``crawl dmoz.org`` subcommand runs the spider for the ``dmoz.org`` domain, y Pay attention to the lines labeled ``[dmoz/dmoz.org]``, which corresponds to our spider identified by the domain "dmoz.org". You can see a log line for each URL defined in ``start_urls``. Because these URLs are the starting ones, they -have no referrers, and this condition is indicated at the end of the log line, -where it says ``from ``. +have no referrers, which is shown at the end of the log line, where it says +``from ``. But more interesting, as our ``parse`` method instructs, two files have been created: *Books* and *Resources*, with the content of both URLs. @@ -158,8 +161,9 @@ Scrapy creates :class:`scrapy.http.Request` objects for each URL in the ``start_urls`` attribute of the Spider, and assigns them the ``parse`` method of the spider as their callback function. -These Requests are scheduled, then executed, and :class:`scrapy.http.Response` -objects are returned to the generator of the Requests. +These Requests are scheduled, then executed, and a :class:`scrapy.http.Response` +objects are returned and then fed to the spider, through the +:meth:`~scrapy.spider.BaseSpider.parse` method. Extracting Items ---------------- @@ -167,61 +171,73 @@ Extracting Items Introduction to Selectors ^^^^^^^^^^^^^^^^^^^^^^^^^ -In order to extract information from web pages Scrapy adopted `XPath -`_, a language for finding information in a XML -document navigating trough its elements and attributes. +There are several ways to extract data from web pages, Scrapy uses a mechanism +based on `XPath`_ expressions called :ref:`XPath selectors `. +For more information about selectors and other extraction mechanisms see the +:ref:`XPath selectors documentation `. -Here are some examples of XPath queries and their corresponding results: +.. _XPath: http://www.w3.org/TR/xpath -* ``/html/head/title``: Will give you the ``title`` node of the document. -* ``/html/head/title/text()``: Will give you the text inside the ``title`` node of the document. -* ``//td``: Will select all the ``td`` elements. -* ``//div[@class="queryMe"]``: Will select all the ``div`` elements with ``class - = queryMe``. +Here are some examples of XPath expressions and their meanings: -This are really simple examples of what you can do with XPath, we strongly -suggest you to follow this `XPath tutorial -`_ before continuing. +* ``/html/head/title``: selects the ```` element, inside the ``<head>`` + element of a HTML document -Scrapy defines a class :class:`~scrapy.xpath.XPathSelector`, that comes in two -flavours, :class:`~scrapy.xpath.HtmlXPatSelector` (for HTML) and -:class:`~scrapy.xpath.XmlXPathSelector` (for XML). In order to use them you -must instantiate the desired class with a :ref:`Response <request-response>` -object. +* ``/html/head/title/text()``: selects the text inside the aforementioned + ``<title>`` element. + +* ``//td``: selects all the ``<td>`` elements + +* ``//div[@class="mine"]``: selects all ``div`` elements which contain an + attribute ``class="mine"`` + +These are just a couple of simple examples of what you can do with XPath, but +XPath expression are indeed much more powerful. To learn more about XPath we +recommend `this XPath tutorial <http://www.w3schools.com/XPath/default.asp>`_. + +For working with XPaths, Scrapy provides a :class:`~scrapy.xpath.XPathSelector` +class, which comes in two flavours, :class:`~scrapy.xpath.HtmlXPatSelector` +(for HTML data) and :class:`~scrapy.xpath.XmlXPathSelector` (for XML data). In +order to use them you must instantiate the desired class with a +:class:`~scrapy.http.Response` object. You can see selectors as objects that represents nodes in the document structure. So, the first instantiated selectors are associated to the root node, or the entire document. -Selectors have three methods: ``x``, ``extract`` and ``re``. +Selectors have three methods (click on the method to see the complete API +documentation). + +* :meth:`~scrapy.xpath.XPathSelector.x`: returns a list of selectors, each of + them representing the nodes selected by the xpath expression given as + argument. + +* :meth:`~scrapy.xpath.XPathSelector.extract`: returns a unicode string with + the data selected by the XPath selector. + +* :meth:`~scrapy.xpath.XPathSelector.re`: returns a list unicode strings + extracted by applying the regular expression given as argument. -* ``x``: returns a list of selectors, each of them representing the nodes - gotten in the xpath expression given as parameter. -* ``extract``: actually extracts the data contained in the node. Does not - receive parameters. -* ``re``: returns a list of results of a regular expression given as parameter. Trying Selectors in the Shell ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -To illustrate the use of Selectors we're going to use the built-in shell of -Scrapy, notice that in order to use this feature you must have IPython (an -extended Python console) installed on your system. +To illustrate the use of Selectors we're going to use the built-in :ref:`Scrapy +shell <topics-shell>`, which also requires IPython (an extended Python console) +installed on your system. To start a shell you must go to the project's top level directory and run:: - ./scrapy-ctl.py shell http://www.dmoz.org/Computers/Programming/Languages/Python/Books/ + python scrapy-ctl.py shell http://www.dmoz.org/Computers/Programming/Languages/Python/Books/ This is what the shell looks like:: [-] Log opened. - Scrapy 0.7.0 - Interactive scraping console + Welcome to Scrapy shell! + Fetching <http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>... - [-] scrapy.management.web.WebConsole starting on 33227 - [-] scrapy.management.telnet.TelnetConsole starting on 42311 - Downloading URL... Done. ------------------------------------------------------------------------------ - Available local variables: + Available Scrapy variables: xxs: <class 'scrapy.xpath.selector.XmlXPathSelector'> url: http://www.dmoz.org/Computers/Programming/Languages/Python/Books/ spider: <class 'dmoz.spiders.dmoz.OpenDirectorySpider'> @@ -229,8 +245,8 @@ This is what the shell looks like:: item: <class 'scrapy.item.models.ScrapedItem'> response: <class 'scrapy.http.response.html.HtmlResponse'> Available commands: - get <url>: Fetches an url and updates all variables. - scrapehelp: Prints this help. + get [url]: Fetch a new URL or re-fetch current Request + shelp: Prints this help. ------------------------------------------------------------------------------ Python 2.6.1 (r261:67515, Dec 7 2008, 08:27:41) Type "copyright", "credits" or "license" for more information. @@ -243,12 +259,13 @@ This is what the shell looks like:: In [1]: -After the shell loads, it will put the result of the request action for the -given URL in a ``response`` variable, so if you enter ``response.body`` the -downloaded data will be printed on the screen. +After the shell loads, you will have the response fetched in a local +``response`` variable, so if you type ``response.body`` you will see the body +of the response, or you can ``response.headers`` to see its headers. -The shell has also instantiated for two selectors with this respose as an -initialization parameter, so let's try them:: +The shell also instantiates two selectors, one for HTML (in the ``hxs`` +variable) and one for XML (in the ``xxs`` variable)with this response. So let's +try them:: In [1]: hxs.x('/html/head/title') Out[1]: [<HtmlXPathSelector (title) xpath=/html/head/title>] @@ -265,16 +282,22 @@ initialization parameter, so let's try them:: In [5]: hxs.x('/html/head/title/text()').re('(\w+):') Out[5]: [u'Computers', u'Programming', u'Languages', u'Python'] -Actually extracting Items -^^^^^^^^^^^^^^^^^^^^^^^^^ +Extracting the data +^^^^^^^^^^^^^^^^^^^ -Now, let's try to extract the sites information from the directory page. +Now, let's try to extract some real information from those pages. -If you do a ``response.body`` in the console, look at the source code of the -page or better yet use Firebug to inspect the page, you'll find that the sites -part of the code is an ``ul`` tag, in fact the *second* ``ul`` tag. +You could type ``response.body`` in the console, and inspect the source code to +figure out the XPaths you need to use. However, inspecting the raw HTML code +there could become a very tedious task. To make this an easier task, you can +use some Firefox extensions like Firebug. For more information see +:ref:`topics-firebug` and :ref:`topics-firefox`. -So we can select each ``li`` item belonging to the sites list with this code:: +After inspecting the page source you'll find that the web sites information +is inside a ``<ul>`` element, in fact the *second* ``<ul>`` element. + +So we can select each ``<li>`` element belonging to the sites list with this +code:: hxs.x('//ul[2]/li') @@ -306,7 +329,6 @@ Let's add this code to our spider:: from scrapy.spider import BaseSpider from scrapy.xpath.selector import HtmlXPathSelector - class DmozSpider(BaseSpider): domain_name = "dmoz.org" start_urls = [ @@ -329,7 +351,7 @@ Let's add this code to our spider:: Now try crawling the dmoz.org domain again and you'll see sites being printed in your output, run:: - ./scrapy-ctl.py crawl dmoz.org + python scrapy-ctl.py crawl dmoz.org Spiders are supposed to return their scraped data in the form of ScrapedItems, so to actually return the data we've scraped so far, the code for our Spider @@ -340,7 +362,6 @@ should be like this:: from dmoz.items import DmozItem - class DmozSpider(BaseSpider): domain_name = "dmoz.org" start_urls = [ @@ -362,29 +383,31 @@ should be like this:: SPIDER = DmozSpider() -Now doing a crawl on the dmoz.org domain yields DmozItems:: +Now doing a crawl on the dmoz.org domain yields ``DmozItem``'s:: [dmoz/dmoz.org] DEBUG: Scraped DmozItem({'title': [u'Text Processing in Python'], 'link': [u'http://gnosis.cx/TPiP/'], 'desc': [u' - By David Mertz; Addison Wesley. Book in progress, full text, ASCII format. Asks for feedback. [author website, Gnosis Software, Inc.]\n']}) in <http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> [dmoz/dmoz.org] DEBUG: Scraped DmozItem({'title': [u'XML Processing with Python'], 'link': [u'http://www.informit.com/store/product.aspx?isbn=0130211192'], 'desc': [u' - By Sean McGrath; Prentice Hall PTR, 2000, ISBN 0130211192, has CD-ROM. Methods to build XML applications fast, Python tutorial, DOM and SAX, new Pyxie open source XML processing library. [Prentice Hall PTR]\n']}) in <http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> -Item Pipeline -============= +Storing the data (using an Item Pipeline) +========================================= -After an item has been scraped by a Spider, it is sent to the Item Pipeline. +After an item has been scraped by a Spider, it is sent to the :ref:`Item +Pipeline <topics-item-pipeline>`. -The Item Pipeline is a set of user written Python classes that implement a -simple method. They receive the Item, do an action upon it (like validating, -checking for duplicates, store the item), and then decide if the Item continues -trough the Pipeline or it's dropped. +The Item Pipeline is a group of user written Python classes that implement a +simple method. They receive an Item and perform an action over it (for example: +validation, checking for duplicates, or storing it in a database), and then +decide if the Item continues through the Pipeline or it's dropped and no longer +processed. -In small projects like this we will use only one Item Pipeline that stores our -Items. +In small projects (like the one on this tutorial) we will use only one Item +Pipeline that just stores our Items. -Like with the Item, a Pipeline placeholder has been set up for you in the -project creation step, it's in ``dmoz/pipelines.py`` and looks like this:: +As with Items, a Pipeline placeholder has been set up for you in the project +creation step, it's in ``dmoz/pipelines.py`` and looks like this:: - # Define yours item pipelines here + # Define your item pipelines here class DmozPipeline(object): def process_item(self, domain, item): @@ -404,5 +427,6 @@ example in a csv file:: Finale ====== -This covers the basics of Scrapy, but they're a lot of features that haven't -been mentioned. They'll be in further tutorials. +This tutorial covers only the basics of Scrapy, but there's a lot of other +features not mentioned here. We recommend you continue reading the section +:ref:`topics-index`. diff --git a/scrapy/trunk/docs/intro/tutorial/index.rst b/scrapy/trunk/docs/intro/tutorial/index.rst deleted file mode 100644 index b799f20ab..000000000 --- a/scrapy/trunk/docs/intro/tutorial/index.rst +++ /dev/null @@ -1,10 +0,0 @@ -======== -Tutorial -======== - -.. toctree:: - - tutorial1 - tutorial2 - tutorial3 - tutorial4 diff --git a/scrapy/trunk/docs/intro/tutorial/tutorial1.rst b/scrapy/trunk/docs/intro/tutorial/tutorial1.rst deleted file mode 100644 index b24a04ce7..000000000 --- a/scrapy/trunk/docs/intro/tutorial/tutorial1.rst +++ /dev/null @@ -1,55 +0,0 @@ -.. _intro-tutorial1: - -====================== -Creating a new project -====================== - -.. highlight:: sh - -In this tutorial, we'll teach you how to scrape http://www.google.com/dirhp -Google's web directory. - -We'll assume that Scrapy is already installed in your system, if not see -:ref:`intro-install`. - -For starting a new project, enter the directory where you'd like your project -to be located, and run:: - - $ scrapy-admin.py startproject googledir - -As long as Scrapy is well installed and the path is set, this will create a -``googledir`` directory with the following contents:: - - googledir/ - scrapy-ctl.py - googledir/ - __init__.py - items.py - pipelines.py - settings.py - spiders/ - __init__.py - templates/ - ... - -These are basically: - -* ``scrapy-ctl.py``: the project's control script. It's used for running the - different tasks (like "genspider", "crawl" and "parse"). We'll talk more - about this later. - -* ``googledir/``: the project's actual python module, you'll import your code from here. - -* ``googledir/items.py``: were you define the different kinds of items you're going to scrape. - -* ``googledir/pipelines.py``: were you define your item pipelines. - -* ``googledir/settings.py``: the project's settings file. - -* ``googledir/spiders/``: directory where you'll later place your spiders. - -* ``googledir/templates/``: directory containing some templates for newly created - spiders, and where you can put your own. - - -Now you can continue with the next part of the tutorial: :ref:`intro-tutorial2`. diff --git a/scrapy/trunk/docs/intro/tutorial/tutorial2.rst b/scrapy/trunk/docs/intro/tutorial/tutorial2.rst deleted file mode 100644 index 25b5e75f8..000000000 --- a/scrapy/trunk/docs/intro/tutorial/tutorial2.rst +++ /dev/null @@ -1,158 +0,0 @@ -.. _intro-tutorial2: - -================ -Our first spider -================ - -Ok, the time to write our first spider has come. Make sure that you're standing -on your project's directory and run:: - - ./scrapy-ctl.py genspider google_directory google.com - -This should create a file called google_directory.py under ``googledir/spiders`` -directory looking like this:: - - # -*- coding: utf8 -*- - import re - - from scrapy.xpath import HtmlXPathSelector - from scrapy.link.extractors import RegexLinkExtractor - from scrapy.contrib.spiders import CrawlSpider, Rule - from googledir.items import GoogledirItem - - class GoogleDirectorySpider(CrawlSpider): - domain_name = 'google.com' - start_urls = ['http://www.google.com/'] - - rules = ( - Rule(RegexLinkExtractor(allow=(r'Items/', )), 'parse_item', follow=True), - ) - - def parse_item(self, response): - i = GoogledirItem() - #xs = HtmlXPathSelector(response) - #i.attribute('site_id', xs.x('//input[@id="sid"]/@value')) - #i.attribute('name', xs.x('//div[@id="name"]')) - #i.attribute('description', xs.x('//div[@id="description"]')) - return [i] - - SPIDER = GoogleDirectorySpider() - -Now, let's explain a bit what this is all about. - -As you may have noticed, the class that represents the spider is -GoogleDirectorySpider, and it inherits from CrawlSpider. - -This means that this spider will crawl over a website given some crawling -rules, and parse the response you need according to your patterns, which are -defined through the "rules" class attribute. - -This attribute is nothing else but a tuple containing Rule objects. Each Rule -defines a specific behaviour the spider will have while crawling the site. - -Rule objects accept the following parameters (the ones between [ ] are optional): - -* ``link_extractor`` - A LinkExtractor instance, which defines the crawling - patterns for this Rule. - -* ``callback`` - A callback to be called for each link extracted matching the - previous link extractor. - -* ``cb_kwargs`` - A dictionary of keyword arguments to be passed to the - provided callback. - -* ``follow`` - A boolean that determines if links are going to be extracted - from responses matching this Rule or not. - -* ``process_links`` - An optional callback for parsing the extracted links. - -In this case, the Rule would mean something like "search for any url containing -the string 'Items/', parse it with the 'parse_item' method, and try to extract -more links from it". - -Now, that's an example, so we must make our own Rule for our own spider. - -But before that, we must set our start_urls to our real entry point (which is -not actually Google's homepage). - -So we replace that line with:: - - start_urls = ['http://www.google.com/dirhp'] - -Now it's the moment to surf that page, and see how can we do to extract data -from it. - -For this task is almost mandatory that you have Firefox FireBug extension, -which allows you to browse through HTML markup in an easy and comfortable way. -Otherwise you'd have to search for tags manually through the body, which can be -*very* tedious. - -.. image:: scrot1.png - -What we see at first sight, is that the directory is divided in categories, -which are also divided in subcategories. - -However, it seems as if there are more subcategories than the ones being shown -in this page, so we'll keep looking... - -.. image:: scrot2.png - -Right, this looks much more interesting. Not only subcategories themselves have -more subcategories, but they have links to websites (which is in fact the -purpose of the directory). - -Now, there's basically one thing to take into account about the previous, and -it's the fact that apparently, categories urls are always of the kind -http://www.google.com/Category/Subcategory/Another_Subcategory (which is not -very distinctive actually, but possible to use). - -So, having said that, a possible rule set for the categories could be:: - - rules = ( - Rule(RegexLinkExtractor(allow=('google.com/[A-Z][a-zA-Z_/]+$', ), ), - 'parse_category', - follow=True, - ), - ) - -Basically, we told our Rule object to extract links that contain the string -'google.com/' plus any capital letter, plus any letter, the '_' character or -the '/'. - -Also, we set our callback 'parse_category' for each of those crawled links, and -decided to extract more links from them with follow=True. - -Until now, our spider would look something like:: - - # -*- coding: utf8 -*- - import re - - from scrapy.xpath import HtmlXPathSelector - from scrapy.link.extractors import RegexLinkExtractor - from scrapy.contrib.spiders import CrawlSpider, Rule - from googledir.items import GoogledirItem - - class GoogleDirectorySpider(CrawlSpider): - domain_name = 'google.com' - start_urls = ['http://www.google.com/dirhp'] - - rules = ( - Rule(RegexLinkExtractor(allow=('google.com/[A-Z][a-zA-Z_/]+$',),), - 'parse_category', - follow=True, - ), - ) - - def parse_category(self, response): - pass - - SPIDER = GoogleDirectorySpider() - -You can try crawling with this little code, by running:: - - ./scrapy-ctl.py crawl google.com - -and it will actually work, altough it won't do any parsing, since -parse_category is not defined, and that's exactly what we're going to do in the -next part of the tutorial: :ref:`intro-tutorial3`. - diff --git a/scrapy/trunk/docs/intro/tutorial/tutorial3.rst b/scrapy/trunk/docs/intro/tutorial/tutorial3.rst deleted file mode 100644 index 6a7e4a225..000000000 --- a/scrapy/trunk/docs/intro/tutorial/tutorial3.rst +++ /dev/null @@ -1,150 +0,0 @@ -.. _intro-tutorial3: - -================= -Scraping our data -================= - -Before going to extraction we need to make a change to our container of scraped -data called ``GoogledirItem`` and defined in ``googledir/items.py`` module:: - - from scrapy.contrib.item import RobustScrapedItem - - class GoogledirItem(RobustScrapedItem): - """Directory website link""" - - ATTRIBUTES = { - 'guid': basestring, - 'name': basestring, - 'url': basestring, - 'description': basestring, - } - -Be sure to inherit from RobustScrapedItem, not ScrapedItem. - -We will now browse a page containing links to websites stored in the directory -(e.g. http://www.google.com/Top/Arts/Awards/) and see how can we extract the -information we need with XPath. - -As I said before, you'll need FireBug for this task. - -.. image:: scrot3.png - -As you can see, this page's markup is not very descriptive (there are no id or -name attributes, or anything that identifies the links uniquely), so the -ranking bars could be a nice reference at the moment of selecting the desired -area with an XPath expression. - -After using FireBug, we can see that each link is inside a ``td`` tag, which is -itself inside a ``tr`` tag that also contains the link's ranking bar (in another -``td``). - -So we could find the ranking bar; then from it, find its parent (the ``tr``), -and then finally, the link's ``td`` (which contains the data we want to -scrape). - -We loaded the page in the Scrapy shell (very useful for doing this), and tried -an XPath expression in order to find the links, which actually worked. - -Basically, that expression would looks for the ranking bar's ``td`` tag: - "find any ``td`` tag who has a descendant tag ``a`` whose ``href`` - attribute contains the string ``#pagerank``" - -and then, the link's ``td`` tag: - "return the ``font`` tag of each following ``td`` sibling that it has" - -Of course, this may not be the only way to get there (usually there are several -expressions that get you to the same place), but it's quite good for this case. - -Another approach could be, for example, to find any ``font`` tags that have that -grey colour of the links, but I prefer to use the first one because it wouldn't -be so strange if there were other tags with the same colour. - -Anyway, having said that, let's make a possible ``parse_category``: - -First, modify the imports section in the spider code to look like this:: - - import re - - from scrapy.xpath import HtmlXPathSelector - from scrapy.link.extractors import RegexLinkExtractor - from scrapy.contrib.spiders import CrawlSpider, Rule - from scrapy.contrib_exp import adaptors - from googledir.items import GoogledirItem - -Then, put this code in the ``parse_category`` method:: - - from scrapy.contrib_exp import adaptors - - def parse_category(self, response): - # The selector we're going to use in order to extract data from the page - hxs = HtmlXPathSelector(response) - - # The path to website links in directory page - links = hxs.x('//td[descendant::a[contains(@href, "#pagerank")]]/following-sibling::td/font') - - # The list of functions to apply to an attribute before assigning its value - adaptor_pipe = [adaptors.extract, adaptors.delist(''), adaptors.strip] - adaptor_map = { - 'name': adaptor_pipe, - 'url': adaptor_pipe, - 'description': adaptor_pipe, - } - - for link in links: - item = GoogledirItem() - item.set_adaptors(adaptor_map) - - item.attribute('name', link.x('a/text()')) - item.attribute('url', link.x('a/@href')) - item.attribute('description', link.x('font[2]/text()')) - yield item - -Okay, more new stuff here :) This time, items! - -Items -^^^^^ - -Items are the objects we use to represent what you scrape (in this case, -links). Basically, there are two important things about items: attributes, and -adaptors. - -Attributes -"""""""""" - -Attributes are nothing else but the places where you store the data you are -extracting, which in this case are, the name of the linked website, its url, -and a description. Now, in most cases, you'll have to do certain modifications -to this data in order to store it (or do whatever you want to do with it), and -this is done through the adaptors. - -Adaptors -"""""""" - -Adaptors are basically a list of functions that receive a value, modify it (or -not), and then return it. In this case we used only two adaptors: - -* ``extract``, which, as you may imagine, extracts data from the XPath nodes - you provide, and returns it as a list. - -* ``delist``, which joins the list that the previous adaptor returned into a - string. This adaptor itself is a class, and this is due to the fact that you - must specify which delimiter will join the list. That's why we put an - instance to this adaptor in the list. - -* ``strip``, which (as you may imagine), does the same as the python strings - strip method. Cleans up extra spaces before and after the provided string. - -In this case, we used the same adaptors for every attribute, because we're -practically doing nothing to the data, just extracting it. But there might be -situations were certain attributes are handled different than others (in fact, -it will happen once you scrape more complicated sites with more complicated -data). - -The rest of the code is quite self-explanatory. The ``attribute`` method sets the -item's attributes, and the items themselves are put into a list that we'll -return to Scrapy's engine. One simple (although important) thing to remember -here is that you must always return a list that contains either items, -requests, or both, but always inside a list. - -So, we're almost done! Let's now check the last part of the tutorial: -:ref:`intro-tutorial4` diff --git a/scrapy/trunk/docs/intro/tutorial/tutorial4.rst b/scrapy/trunk/docs/intro/tutorial/tutorial4.rst deleted file mode 100644 index 4c066f545..000000000 --- a/scrapy/trunk/docs/intro/tutorial/tutorial4.rst +++ /dev/null @@ -1,78 +0,0 @@ -.. _intro-tutorial4: - -================= -Finishing the job -================= - -Well, we've got our project, our spider, and our scraped items. -What to do next? - -It actually depends on what you want to do with the scraped data. In this -case, we'll imagine that we want to save this data for storing it in a db -later, or just to keep it there. - -To make it simple, we'll export the scraped items to a CSV file by making use -of a useful function that Scrapy brings: ``items_to_csv``. This simple function -takes a file descriptor/filename, and a list of items, and writes their -attributes to that file, in CSV format. - -.. highlight:: python - -Let's see how would our spider end up looking like after applying this change:: - - # -*- coding: utf8 -*- - import re - - from scrapy.xpath import HtmlXPathSelector - from scrapy.link.extractors import RegexLinkExtractor - from scrapy.contrib.spiders import CrawlSpider, Rule - from scrapy.contrib_exp import adaptors - from scrapy.utils.misc import items_to_csv - from googledir.items import GoogledirItem - - class GoogleDirectorySpider(CrawlSpider): - domain_name = 'google.com' - start_urls = ['http://www.google.com/dirhp'] - - rules = ( - Rule(RegexLinkExtractor(allow=('google.com/[A-Z][a-zA-Z_/]+$',),), - 'parse_category', - follow=True, - ), - ) - csv_file = open('scraped_items.csv', 'ab+') - - def parse_category(self, response): - # The selector we're going to use in order to extract data from the page - hxs = HtmlXPathSelector(response) - - # The path to website links in directory page - links = hxs.x('//td[descendant::a[contains(@href, "#pagerank")]]/following-sibling::td/font') - - # The list of functions to apply to an attribute before assigning its value - adaptor_pipe = [adaptors.extract, adaptors.delist(''), adaptors.strip] - adaptor_map = { - 'name': adaptor_pipe, - 'url': adaptor_pipe, - 'description': adaptor_pipe, - } - - for link in links: - item = GoogledirItem() - item.set_adaptors(adaptor_map) - - item.attribute('name', link.x('a/text()')) - item.attribute('url', link.x('a/@href')) - item.attribute('description', link.x('font[2]/text()')) - items_to_csv(self.csv_file, [item]) - yield item - -SPIDER = GoogleDirectorySpider() - -With this code, our spider will crawl over Google's directory, and save each -link's name, description, and url to a file called 'scraped_items.csv':: - - ./scrapy-ctl.py crawl google.com - -This is the end of the tutorial. If you'd like to know more about Scrapy and -its use, please read the rest of the documentation. diff --git a/scrapy/trunk/docs/misc/api-stability.rst b/scrapy/trunk/docs/misc/api-stability.rst index cd1d012e0..c43fcd3cb 100644 --- a/scrapy/trunk/docs/misc/api-stability.rst +++ b/scrapy/trunk/docs/misc/api-stability.rst @@ -23,37 +23,12 @@ relevant. API Stability ============= -Methods or functions that start with a single ``_`` are private and should -never be relied as stable. Besides those, the plan is to stabilize and document -the entire API, as we approach the 1.0 release. In the meantime, you'll find -here a list of the APIs that we consider already stable. +Methods or functions that start with a single dash (``_``) are private and +should never be relied as stable. Besides those, the plan is to stabilize and +document the entire API, as we approach the 1.0 release. In the meantime, +you'll find here a list of the APIs that we consider already stable. Also, keep in mind that stable doesn't mean complete: stable APIs could grow new methods or functionality but the existing methods should keep working the same way. -Stable APIs ------------ - -The APIs listed here should keep working as documented between minor versions. - -* :ref:`topics-items` -* :ref:`topics-selectors` -* :ref:`topics-item-pipeline` -* :ref:`topics-downloader-middleware` -* :ref:`topics-spider-middleware` -* :ref:`topics-settings` - -Almost stable APIs ------------------- - -The APIs listed here may suffer minor changes between minor versions. - -Unstable APIs -------------- - -These APIs may suffer major changes or be removed completely on the next minor -version release. - -* :ref:`topics-adaptors` - diff --git a/scrapy/trunk/docs/proposed/index.rst b/scrapy/trunk/docs/proposed/index.rst index 7989dedd7..2fdf8134b 100644 --- a/scrapy/trunk/docs/proposed/index.rst +++ b/scrapy/trunk/docs/proposed/index.rst @@ -18,7 +18,3 @@ Proposed documentation introduction -.. toctree:: - :maxdepth: 1 - - tutorial diff --git a/scrapy/trunk/docs/ref/index.rst b/scrapy/trunk/docs/ref/index.rst index 0a5746f05..c8516067c 100644 --- a/scrapy/trunk/docs/ref/index.rst +++ b/scrapy/trunk/docs/ref/index.rst @@ -3,10 +3,10 @@ API Reference ============= -This section documents the API of Scrapy |version|. For more information see :ref:`misc-api-stability`. +This section documents the Scrapy |version| API. For more information see :ref:`misc-api-stability`. .. toctree:: - :maxdepth: 1 + :maxdepth: 2 spiders selectors diff --git a/scrapy/trunk/docs/ref/selectors.rst b/scrapy/trunk/docs/ref/selectors.rst index 3d84fe49b..9ebbb3110 100644 --- a/scrapy/trunk/docs/ref/selectors.rst +++ b/scrapy/trunk/docs/ref/selectors.rst @@ -1,8 +1,8 @@ .. _ref-selectors: -============= -Selectors API -============= +=================== +XPath Selectors API +=================== .. module:: scrapy.xpath :synopsis: XPath selectors classes diff --git a/scrapy/trunk/docs/ref/settings.rst b/scrapy/trunk/docs/ref/settings.rst index ed000f00e..5773e903d 100644 --- a/scrapy/trunk/docs/ref/settings.rst +++ b/scrapy/trunk/docs/ref/settings.rst @@ -18,7 +18,9 @@ ADAPTORS_DEBUG Default: ``False`` -Enable debug mode for adaptors. +Enable debug mode for adaptors. + +See :ref:`topics-adaptors`. .. setting:: BOT_NAME diff --git a/scrapy/trunk/docs/intro/tutorial/scrot1.png b/scrapy/trunk/docs/topics/_images/firebug1.png similarity index 100% rename from scrapy/trunk/docs/intro/tutorial/scrot1.png rename to scrapy/trunk/docs/topics/_images/firebug1.png diff --git a/scrapy/trunk/docs/intro/tutorial/scrot2.png b/scrapy/trunk/docs/topics/_images/firebug2.png similarity index 100% rename from scrapy/trunk/docs/intro/tutorial/scrot2.png rename to scrapy/trunk/docs/topics/_images/firebug2.png diff --git a/scrapy/trunk/docs/intro/tutorial/scrot3.png b/scrapy/trunk/docs/topics/_images/firebug3.png similarity index 100% rename from scrapy/trunk/docs/intro/tutorial/scrot3.png rename to scrapy/trunk/docs/topics/_images/firebug3.png diff --git a/scrapy/trunk/docs/topics/adaptors.rst b/scrapy/trunk/docs/topics/adaptors.rst index 8781f95f6..89be0cefa 100644 --- a/scrapy/trunk/docs/topics/adaptors.rst +++ b/scrapy/trunk/docs/topics/adaptors.rst @@ -1,8 +1,8 @@ .. _topics-adaptors: -======== -Adaptors -======== +======================= +Adaptors (experimental) +======================= .. warning:: diff --git a/scrapy/trunk/docs/topics/architecture.rst b/scrapy/trunk/docs/topics/architecture.rst index 9fb9c0a02..702cc00f6 100644 --- a/scrapy/trunk/docs/topics/architecture.rst +++ b/scrapy/trunk/docs/topics/architecture.rst @@ -1,8 +1,8 @@ .. _topics-architecture: -============ -Architecture -============ +===================== +Architecture overview +===================== This document describes the architecture of Scrapy and how their components interact. diff --git a/scrapy/trunk/docs/topics/firebug.rst b/scrapy/trunk/docs/topics/firebug.rst new file mode 100644 index 000000000..c62f9530f --- /dev/null +++ b/scrapy/trunk/docs/topics/firebug.rst @@ -0,0 +1,154 @@ +.. _topics-firebug: + +========================== +Using Firebug for scraping +========================== + +Introduction +============ + +This document explains how to use `Firebug`_ (a Firefox add-on) to make the +scraping process easier and more fun. For other useful Firefox add-ons see +:ref:`topics-firefox`. + +In this example, we'll show how to use `Firebug`_ to scrape data from the +`Google Directory`_, which contains the same data as the `Open Directory +Project`_ used in the :ref:`tutorial <intro-tutorial>` but with a different +face. + +.. _Firebug: http://getfirebug.com +.. _Google Directory: http://www.google.com/dirhp +.. _Open Directory Project: http://www.dmoz.org + +Firebug comes with a very useful feature called `Inspect Element`_ which allows +you to inspect the HTML code of the different page elements just by hovering +your mouse over them. Otherwise you would have to search for the tags manually +through the HTML body which can be a very tedious task. + +.. _Inspect Element: http://www.youtube.com/watch?v=-pT_pDe54aA + +In the following screenshot you can see the `Inspect Element`_ tool in action. + +.. image:: _images/firebug1.png + :width: 913 + :height: 600 + :alt: Inspecting elements with Firebug + +At first sight, we can see that the directory is divided in categories, which +are also divided in subcategories. + +However, it seems as if there are more subcategories than the ones being shown +in this page, so we'll keep looking: + +.. image:: _images/firebug2.png + :width: 819 + :height: 629 + :alt: Inspecting elements with Firebug + +As expected the subcategories contain links to other subcategories, and also +links to actual websites, which is the purpose of the directory. + +Getting links to follow +======================= + +By looking at the category URLs we can see they share a pattern: + + http://www.google.com/Category/Subcategory/Another_Subcategory + +Once we know that, we are able to construct a regular expression to follow +those links. For example, the following one:: + + google.com/[A-Z][a-zA-Z_/]+$ + +So, based on that regular expression we can create the first crawling rule:: + + Rule(RegexLinkExtractor(allow='google.com/[A-Z][a-zA-Z_/]+$', ), + 'parse_category', + follow=True, + ), + +The :class:`~scrapy.contrib.spiders.Rule` object instructs +:class:`~scrapy.contrib.spiders.CrawlSpider` based spiders how to follow the +category links. ``parse_category`` will be a method of the spider which will +process and extract data from those pages. + +This is how the spider would look so far:: + + from scrapy.link.extractors import RegexLinkExtractor + from scrapy.contrib.spiders import CrawlSpider, Rule + + class GoogleDirectorySpider(CrawlSpider): + domain_name = 'google.com' + start_urls = ['http://www.google.com/dirhp'] + + rules = ( + Rule(RegexLinkExtractor(allow='google.com/[A-Z][a-zA-Z_/]+$'), + 'parse_category', follow=True, + ), + ) + + def parse_category(self, response): + # write the category page data extraction code here + pass + + SPIDER = GoogleDirectorySpider() + + +Extracting the data +=================== + +Now we're gonna write the code to extract data from those pages. + +With the help of Firebug, we'll take a look at some page containing links to +websites (say http://www.google.com/Top/Arts/Awards/) and find out how we can +extract those links using :ref:`XPath selectors <topics-selectors>`. We'll also +use the :ref:`Scrapy shell <topics-shell>` to test those XPath's and make sure +they work as we expect. + +.. image:: _images/firebug3.png + :width: 965 + :height: 751 + :alt: Inspecting elements with Firebug + +As you can see, the page markup is not very descriptive: the elements don't +contain ``id``, ``class`` or any attribute that clearly identifies them, so +we''ll use the ranking bars as a reference point to select the data to extract +when we construct our XPaths. + +After using FireBug, we can see that each link is inside a ``td`` tag, which is +itself inside a ``tr`` tag that also contains the link's ranking bar (in +another ``td``). + +So we can select the ranking bar, then find its parent (the ``tr``), and then +finally, the link's ``td`` (which contains the data we want to scrape). + +This results in the following XPath:: + + //td[descendant::a[contains(@href, "#pagerank")]]/following-sibling::td//a + +It's important to use the :ref:`Scrapy shell <topics-shell>` to test these +complex XPath expressions and make sure they work as expected. + +Basically, that expression will look for the ranking bar's ``td`` element, and +then select any ``td`` element who has a descendant ``a`` element whose +``href`` attribute contains the string ``#pagerank``" + +Of course, this is not the only XPath, and maybe not the simpler one to select +that data. Another approach could be, for example, to find any ``font`` tags +that have that grey colour of the links, + +Finally, we can write our ``parse_category()`` method:: + + def parse_category(self, response): + hxs = HtmlXPathSelector(response) + + # The path to website links in directory page + links = hxs.x('//td[descendant::a[contains(@href, "#pagerank")]]/following-sibling::td/font') + + for link in links: + item = ScrapedItem() + item.name = link.x('a/text()').extract() + item.url = link.x('a/@href').extract() + item.description = link.x('font[2]/text()').extract() + yield item + diff --git a/scrapy/trunk/docs/topics/firefox.rst b/scrapy/trunk/docs/topics/firefox.rst new file mode 100644 index 000000000..0d1a06f4e --- /dev/null +++ b/scrapy/trunk/docs/topics/firefox.rst @@ -0,0 +1,52 @@ +.. _topics-firefox: + +=================================== +Useful Firefox add-ons for scraping +=================================== + +Here is a list of useful Firefox add-ons you can use to make the scraping +process easier, faster and more comfortable. + +Firebug +======= + +`Firebug`_ is a widely known tool among web developers and it's also very +useful for scraping. In particular, its `Inspect Element`_ feature comes very +handy when you need to construct the XPaths for extracting data because it +allows you to view the HTML code of each page element while moving your mouse +over them. + +See :ref:`topics-firebug` for a detailed guide on how to use Firebug with +Scrapy. + +XPather +======= + +`XPather`_ allows you to test XPath expressions directly on the pages. + +XPath Checker +============= + +`XPath Checker`_ is another Firefox add-on for testing XPaths on your pages. + +Tamper Data +=========== + +`Tamper Data`_ is a Firefox add-on which allows you to view and modify the HTTP +request headers sent by Firefox. Firebug also allows to view HTTP headers, but +not to modify them. + +Firecookie +========== + +`Firecookie`_ makes it easier to view and manage cookies. You can use this +extension to create a new cookie, delete existing cookies, see list of cookies +for current site, manage cookies permissions and a lot more. + +.. _Firebug: http://getfirebug.com +.. _Inspect Element: http://www.youtube.com/watch?v=-pT_pDe54aA +.. _XPather: https://addons.mozilla.org/firefox/addon/1192 +.. _XPath Checker: https://addons.mozilla.org/firefox/addon/1095 +.. _Tamper Data: http://addons.mozilla.org/firefox/addon/966 +.. _Firecookie: https://addons.mozilla.org/firefox/addon/6683 + diff --git a/scrapy/trunk/docs/topics/index.rst b/scrapy/trunk/docs/topics/index.rst index eb8e8fc6d..abe9b18be 100644 --- a/scrapy/trunk/docs/topics/index.rst +++ b/scrapy/trunk/docs/topics/index.rst @@ -3,7 +3,7 @@ Using Scrapy ============ -This section introduces all key concepts of Scrapy. +This section describes all key concepts of Scrapy. .. toctree:: :maxdepth: 1 @@ -14,7 +14,6 @@ This section introduces all key concepts of Scrapy. selectors link-extractors items - adaptors item-pipeline downloader-middleware spider-middleware @@ -23,3 +22,5 @@ This section introduces all key concepts of Scrapy. stats webconsole robotstxt + firebug + firefox diff --git a/scrapy/trunk/docs/topics/selectors.rst b/scrapy/trunk/docs/topics/selectors.rst index cf59d7200..87bc3f84a 100644 --- a/scrapy/trunk/docs/topics/selectors.rst +++ b/scrapy/trunk/docs/topics/selectors.rst @@ -1,14 +1,14 @@ .. _topics-selectors: -========= -Selectors -========= +=============== +XPath Selectors +=============== Introduction ------------ When you're scraping web pages, the most common task you need to perform is -extract data from the HTML source. There are several libraries available to +to extract data from the HTML source. There are several libraries available to achieve this: * `BeautifulSoup`_ is a very popular screen scraping library among Python @@ -20,12 +20,12 @@ achieve this: API based on `ElementTree`_ (which is not part of the Python standard library). -Scrapy comes with its own mechanism for mechanism for extracting data. They're -called selectors, because they "select" certain parts of the HTML document -specified by XPaths. +Scrapy comes with its own mechanism for extracting data. They're called XPath +selectors (or just "selectors", for short) because they "select" certain parts +of the HTML document specified by `XPath`_ expressions. -`XPath`_ is a language for selecting nodes in XML documents, and it's also -applicable for HTML documents. +`XPath`_ is a language for selecting nodes in XML documents, which can be used +to with HTML. Both `lxml`_ and Scrapy Selectors are built over the `libxml2`_ library, which means they're very similar in speed and parsing accuracy. diff --git a/scrapy/trunk/docs/topics/shell.rst b/scrapy/trunk/docs/topics/shell.rst index ed332673f..7f746d540 100644 --- a/scrapy/trunk/docs/topics/shell.rst +++ b/scrapy/trunk/docs/topics/shell.rst @@ -1,8 +1,8 @@ .. _topics-shell: -===== -Shell -===== +================ +The Scrapy shell +================ The Scrapy shell is an interactive shell where you can try and debug your scraping code very quickly, without having to run the spider. It's meant to be @@ -109,7 +109,7 @@ shell works. :: - scrapy-ctl.py shell http://scrapy.org + python scrapy-ctl.py shell http://scrapy.org 2009-04-02 16:56:22-0300 [-] Log opened. Welcome to Scrapy shell! @@ -125,8 +125,8 @@ shell works. item: <class 'myproject.models.Item'> response: <http://scrapy.org> Available commands: - get <url>: Fetches an url and updates all variables. - scrapehelp: Prints this help. + get [url]: Fetch a new URL or re-fetch current Request + shelp: Prints this help. ------------------------------------------------------------ Python 2.5.2 (r252:60911, Oct 5 2008, 19:29:17) Type "copyright", "credits" or "license" for more information.