From 279b9ac40d7e38713f74a6a74e123475577f1b88 Mon Sep 17 00:00:00 2001 From: Pablo Hoffman Date: Mon, 26 Jan 2009 23:22:53 +0000 Subject: [PATCH] updated adaptors docs to reflect its inestability and improved the ReST formatting (80 column lines, no ugly pipes at the beginnig of lines) --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40775 --- scrapy/trunk/docs/misc/api-stability.rst | 3 +- scrapy/trunk/docs/topics/adaptors.rst | 140 +++++++++++++++-------- 2 files changed, 92 insertions(+), 51 deletions(-) diff --git a/scrapy/trunk/docs/misc/api-stability.rst b/scrapy/trunk/docs/misc/api-stability.rst index 65476e645..cd1d012e0 100644 --- a/scrapy/trunk/docs/misc/api-stability.rst +++ b/scrapy/trunk/docs/misc/api-stability.rst @@ -49,12 +49,11 @@ Almost stable APIs The APIs listed here may suffer minor changes between minor versions. -* :ref:`topics-adaptors` - Unstable APIs ------------- These APIs may suffer major changes or be removed completely on the next minor version release. +* :ref:`topics-adaptors` diff --git a/scrapy/trunk/docs/topics/adaptors.rst b/scrapy/trunk/docs/topics/adaptors.rst index 5523283ed..96306ed33 100644 --- a/scrapy/trunk/docs/topics/adaptors.rst +++ b/scrapy/trunk/docs/topics/adaptors.rst @@ -4,30 +4,45 @@ Adaptors ======== +.. warning:: + + Adaptors are an experimental feature of Scrapy, which mean its API is not + yet stable and could suffer minor changes before the next stable release. + Quick overview ============== -| Adaptors are basically functions that receive one value (advanced adaptors may receive more, but we'll see that later), modify it, and return - a new value. -| In order to adapt our scraped data we can use an adaptor pipeline for each of the item's attributes. -| Adaptor pipelines are nothing else but a list of adaptors which will be iterated at the moment of assigning an attribute, calling each adaptor - and passing the values from one to each other. +Adaptors are basically functions that receive one value (advanced adaptors may +receive more, but we'll see that later), modify it, and return a new value. + +In order to adapt our scraped data we can use an adaptor pipeline for each of the item's attributes. + +Adaptor pipelines are nothing else but a list of adaptors which will be +iterated at the moment of assigning an attribute, calling each adaptor and +passing the values from one to each other. Example use =========== -| The most common example use of adaptors appears when parsing HTML pages. To do this, we normally use XPathSelectors to retrieve data, which need to be extracted some - way, and in most cases, filtered some way. -| You could extract them yourself, as well as doing any kind of adaptation before assigning, but the idea of adaptor pipelines is to simplify this task, and the spider's code. +The most common example use of adaptors appears when parsing HTML pages. To do +this, we normally use XPathSelectors to retrieve data, which need to be +extracted some way, and in most cases, filtered some way. -Let's imagine, for example, that you want to scrape information from pages as the following: +You could extract them yourself, as well as doing any kind of adaptation before +assigning, but the idea of adaptor pipelines is to simplify this task, and the +spider's code. + +Let's imagine, for example, that you want to scrape information from pages as +the following: .. literalinclude:: ../_static/items_adaptors-sample1.html :language: html -| In this case, you'd have to scrape some products information, like their manufacturer, name, and price. -| We'll put this information inside ScrapedItems and attach them some adaptors to process our data better. -| You can test yourself with this page, since it actually exists `here <../_static/items_adaptors-sample1.html>`_ +In this case, you'd have to scrape some products information, like their +manufacturer, name, and price. We'll put this information inside ScrapedItems +and attach them some adaptors to process our data better. You can test yourself +with this page, since it actually exists `here +<../_static/items_adaptors-sample1.html>`_ .. highlight:: sh @@ -51,20 +66,26 @@ And then let's try to find the products and scrape an item manually. Something l >>> item.manufacturer [] -| Okay, what we did here was creating an item, and setting its 'manufacturer' attribute by using the selectors we already had for each product row. -| As you can see, we didn't apply the extract method to the selector, so the data stored in the product was exactly the same that our selector's call to the ``x`` method - returns; another selector. +Okay, what we did here was creating an item, and setting its 'manufacturer' +attribute by using the selectors we already had for each product row. + +As you can see, we didn't apply the extract method to the selector, so the data +stored in the product was exactly the same that our selector's call to the +``x`` method returns; another selector. + We could extract the information before setting it, but we'll use adaptors instead: - >>> from scrapy.contrib import adaptors + >>> from scrapy.contrib_exp import adaptors >>> item = ScrapedItem() >>> item.add_adaptor('manufacturer', adaptors.extract) >>> item.attribute('manufacturer', product_rows[0].x('td[@class="prod_attrib"][1]/text()')) >>> item.manufacturer [u"Bill & Ted's Farm"] -| Better now, right? At least we have some readable data :) -| Although we'd probably want to remove those entities, and to store the information as a string... +Better now, right? At least we have some readable data :) + +Although we'd probably want to remove those entities, and to store the +information as a string... >>> item = ScrapedItem() >>> item.set_attrib_adaptors('manufacturer', [ @@ -75,11 +96,13 @@ And then let's try to find the products and scrape an item manually. Something l >>> item.manufacturer u"Bill & Ted's Farm" -| Cool, now that looks like something that can be stored correctly. -| However, we must look at the rest of the attributes too. -| At first sight, it looks like at least the name and the description could use the same adaptors as the manufacturer, since they're all a simple text. +Cool, now that looks like something that can be stored correctly. However, we +must look at the rest of the attributes too. At first sight, it looks like at +least the name and the description could use the same adaptors as the +manufacturer, since they're all a simple text. -The weight and the price could get a better parsing though; at least to convert them to real decimals. Let's experiment a bit more:: +The weight and the price could get a better parsing though; at least to convert +them to real decimals. Let's experiment a bit more:: >>> from decimal import Decimal # We'll use Decimal objects for storing the price and weight @@ -89,9 +112,13 @@ The weight and the price could get a better parsing though; at least to convert >>> item.price Decimal('300') -| In this case it wasn't necessary to use the extract adaptor, because applying the ``re`` method over a selector already extracts the resulting content. -| Now, the price was quite easy, but the weight is a bit tricky, for the simple fact that it has more than one weighing unit. In this case, we'll have - to write our own little adaptor that takes care of parsing a string and returning a Decimal according to its unit. +In this case it wasn't necessary to use the extract adaptor, because applying +the ``re`` method over a selector already extracts the resulting content. + +Now, the price was quite easy, but the weight is a bit tricky, for the simple +fact that it has more than one weighing unit. In this case, we'll have to write +our own little adaptor that takes care of parsing a string and returning a +Decimal according to its unit. Something like this could work, although it's very primitive:: @@ -137,7 +164,7 @@ Ok, done! Let's now sum this up into a spider:: from decimal import Decimal from scrapy.item import ScrapedItem - from scrapy.contrib import adaptors + from scrapy.contrib_exp import adaptors from scrapy.contrib.spiders import CrawlSpider, Rule from scrapy.xpath.selector import HtmlXPathSelector from scrapy.link.extractors import RegexLinkExtractor @@ -193,39 +220,53 @@ Ok, done! Let's now sum this up into a spider:: SPIDER = MySpider() -| Basically this spider looks for the product rows in the page, creates an item for each of them, attaches them some adaptors, and fills their attributes in. +Basically this spider looks for the product rows in the page, creates an item +for each of them, attaches them some adaptors, and fills their attributes in. Scraping the sample page with this code would give us these items:: ScrapedItem({u"name": u"Bananas", u"manufacturer": u"Bill & Ted's farm", u"description": "Delicious fruit", u"weight": Decimal("2000"), u"price": Decimal("300")}) ScrapedItem({u"name": u"Apple pie", u"manufacturer": u"Grandma's", u"description": "Grandma's best dish", u"weight": Decimal("250"), u"price": Decimal("200")}) -There could be more parsing done here through adaptors, like parsing different price currencies, or more advanced weight parsing (this one was very, very, simple and buggy). -Nevertheless, I hope that this was useful as an example use of adaptors. +There could be more parsing done here through adaptors, like parsing different +price currencies, or more advanced weight parsing (this one was very, very, +simple and buggy). Nevertheless, I hope that this was useful as an example use +of adaptors. More complex adaptors ===================== -| Okay, what happens when you have adaptors that receive extra parameters in order to modify their behaviour? -| For example, going back to money, what if you have a function capable of converting from one currency to another one by receiving three parameters: the value, its currency, - and the currency to convert to? -| Well, don't worry, Scrapy can handle this situation and use this adaptor too, and in fact, its something quite easy to do. -| -| Basically the difference between a regular adaptor and a "complex" one (I call them complex just for the fact that the others are *very* simple), is that the first one - always receives and returns a single value, while the others may receive more, passed by you to the item's ``attribute`` method. -| Whenever this method (``attribute``) receives any extra parameters (apart from the value to set, ``override``, and ``add``), it automatically passes them to any adaptor - that receives a parameter called ``adaptor_args`` in a dictionary. -| This adaptor should read the dictionary and check if it has received any parameters there. -| -| Notice that you should be careful with the naming of these parameters in order to not generate confusion between adaptors, because every adaptor (that has the ``adaptor_args`` - parameter, of course) will receive the same dict of keywords, containing keywords that are destined for itself, and some that aren't. -| For example, it's not very wise to use a parameter called 'encoding' unless you'd like all the adaptors to receive it; otherwise it'd be better to use the adaptor's name - as a prefix to the parameter, just to make it clearer. -| -| Let's now see the currency example, now written in code. +Okay, what happens when you have adaptors that receive extra parameters in +order to modify their behaviour? For example, going back to money, what if you +have a function capable of converting from one currency to another one by +receiving three parameters: the value, its currency, and the currency to +convert to?. Well, don't worry, Scrapy can handle this situation and use this +adaptor too, and in fact, its something quite easy to do. -First, we'll define a class called Currency and a dictionary, in order to store some currencies and their exchange rate:: +Basically the difference between a regular adaptor and a "complex" one (I call +them complex just for the fact that the others are *very* simple), is that the +first one always receives and returns a single value, while the others may +receive more, passed by you to the item's ``attribute`` method. Whenever this +method (``attribute``) receives any extra parameters (apart from the value to +set, ``override``, and ``add``), it automatically passes them to any adaptor +that receives a parameter called ``adaptor_args`` in a dictionary. This +adaptor should read the dictionary and check if it has received any parameters +there. + +Notice that you should be careful with the naming of these parameters in order +to not generate confusion between adaptors, because every adaptor (that has the +``adaptor_args`` parameter, of course) will receive the same dict of keywords, +containing keywords that are destined for itself, and some that aren't. + +For example, it's not very wise to use a parameter called 'encoding' unless +you'd like all the adaptors to receive it; otherwise it'd be better to use the +adaptor's name as a prefix to the parameter, just to make it clearer. + +Let's now see the currency example, now written in code. + +First, we'll define a class called Currency and a dictionary, in order to store +some currencies and their exchange rate:: class Currency(object): def __init__(self, from_dollars, to_dollars): @@ -252,7 +293,8 @@ First, we'll define a class called Currency and a dictionary, in order to store 'pound': Currency(from_dollars=Decimal('0.6547'), to_dollars=Decimal('1.5274')), } -Ok, now that we've got some exchange information, the only thing missing is the adaptor that makes use of it:: +Ok, now that we've got some exchange information, the only thing missing is the +adaptor that makes use of it:: def exchange_adaptor(value, adaptor_args): currency_from = adaptor_args.get('currency_from', 'dollar') @@ -264,7 +306,7 @@ Ok, now that we've got some exchange information, the only thing missing is the else: raise Exception('Unsupported currencies were specified') -And finally, testing it!:: +And finally, testing it! :: >>> item = ScrapedItem() >>> item.set_attrib_adaptors('price', [ Decimal, exchange_adaptor ])