diff --git a/scrapy/trunk/docs/topics/adaptors.rst b/scrapy/trunk/docs/topics/adaptors.rst index f89b619a7..108956135 100644 --- a/scrapy/trunk/docs/topics/adaptors.rst +++ b/scrapy/trunk/docs/topics/adaptors.rst @@ -12,307 +12,135 @@ Adaptors Quick overview ============== -Adaptors are basically functions that receive one value (advanced adaptors may -receive more, but we'll see that later), modify it, and return a new value. +Scrapy's adaptors are a nice feature attached to RobustScrapedItems that allow you +to easily modify (adapt to your needs) any kind of information you want to put in your items at assignation time. -In order to adapt our scraped data we can use an adaptor pipeline for each of the item's attributes. +The following diagram shows the data flow from the moment you call the `attribute` method until the attribute is +actually set. -Adaptor pipelines are nothing else but a list of adaptors which will be -iterated at the moment of assigning an attribute, calling each adaptor and -passing the values from one to each other. +.. image:: _images/adaptors_diagram.png -Example use -=========== +As you can see, adaptor pipelines are executed in tree form; which means that, for each of the values you pass to +the `attribute` method, the first adaptor will be applied. Then, for each of the resulting values of the first adaptor, +the second adaptor will be called, and so on. +This process will end up with a list of adapted values, which may contain zero, one, or many values. -The most common example use of adaptors appears when parsing HTML pages. To do -this, we normally use XPathSelectors to retrieve data, which need to be -extracted some way, and in most cases, filtered some way. +In case the attribute is a single-valued (this is defined in the item's ATTRIBUTES dictionary), the first element of this +list will be set, unless you call the `attribute` method with the add parameter as True, in which case the item's method +`_add_single_attributes` will be called with the attribute's name, type, and the list of attributes to join as parameters. +By default, this method raises NotImplementedError, so you should override it in your items in order to join any kind of objects. -You could extract them yourself, as well as doing any kind of adaptation before -assigning, but the idea of adaptor pipelines is to simplify this task, and the -spider's code. +If the attribute is a multivalued, the resulting list will be set to the item as is, unless you use -again- add=True, +in which case the list of already-existing values (if any) will be extended with the new one. -Let's imagine, for example, that you want to scrape information from pages as -the following: +Adaptor Pipelines +================= -.. literalinclude:: ../_static/items_adaptors-sample1.html - :language: html +.. class:: AdaptorPipe(adaptors=None) -In this case, you'd have to scrape some products information, like their -manufacturer, name, and price. We'll put this information inside ScrapedItems -and attach them some adaptors to process our data better. You can test yourself -with this page, since it actually exists `here -<../_static/items_adaptors-sample1.html>`_ + An instance of this class represents an adaptor pipeline to be set for adapting a certain + item's attribute. + It provides some useful methods for adding/removing adaptors, and takes care of executing them properly. + Usually this class is not used directly, since the items already provide ways to handle adaptors without + having to manage AdaptorPipes. -.. highlight:: sh + :param adaptors: A list of callables to be added as adaptors at instancing time. -Open it in a Scrapy shell by doing:: + Methods: - ./scrapy-ctl.py shell 'http://doc.scrapy.org/_static/items_adaptors-sample1.html' + .. method:: add_adaptor(adaptor, position=None) -.. highlight:: python + This method is used for adding adaptors to the pipeline given a certain position. -And then let's try to find the products and scrape an item manually. Something like:: + :param adaptor: Any callable that works as an adaptor + :param position: An integer meaning the position in which the adaptor will be inserted. If it's None + the adaptor will be appended at the end of the pipeline. - >>> product_rows = hxs.x('//tr[child::td[@class="prod_attrib"]]') - >>> product_rows - [, - ] +Usage +===== - # Now we'll play trying to see how we scrape the first product - >>> from scrapy.item import ScrapedItem - >>> item = ScrapedItem() - >>> item.attribute('manufacturer', product_rows[0].x('td[@class="prod_attrib"][1]/text()')) - >>> item.manufacturer - [] +As it was previously said, in order to use adaptor pipelines you must inherit your items from the RobustScrapedItem class. +If you don't know anything about these items, read the :ref:`topics-items` reference first. -Okay, what we did here was creating an item, and setting its 'manufacturer' -attribute by using the selectors we already had for each product row. +Once you've created your own item class (inherited from RobustScrapedItem) with the attributes you're going to use, +you have to add adaptor pipelines to each attribute you'd like to adapt data for. +For doing so, RobustScrapedItems provide some useful methods (`set_adaptors`, `set_attrib_adaptors`, and more), which are +also described in its reference. -As you can see, we didn't apply the extract method to the selector, so the data -stored in the product was exactly the same that our selector's call to the -``x`` method returns; another selector. - - We could extract the information before setting it, but we'll use adaptors instead: - - >>> from scrapy.contrib_exp import adaptors - >>> item = ScrapedItem() - >>> item.add_adaptor('manufacturer', adaptors.extract) - >>> item.attribute('manufacturer', product_rows[0].x('td[@class="prod_attrib"][1]/text()')) - >>> item.manufacturer - [u"Bill & Ted's Farm"] - -Better now, right? At least we have some readable data :) - -Although we'd probably want to remove those entities, and to store the -information as a string... - - >>> item = ScrapedItem() - >>> item.set_attrib_adaptors('manufacturer', [ - adaptors.extract, - adaptors.Unquote(), - adaptors.Delist() ]) - >>> item.attribute('manufacturer', product_rows[0].x('td[@class="prod_attrib"][1]/text()')) - >>> item.manufacturer - u"Bill & Ted's Farm" - -Cool, now that looks like something that can be stored correctly. However, we -must look at the rest of the attributes too. At first sight, it looks like at -least the name and the description could use the same adaptors as the -manufacturer, since they're all a simple text. - -The weight and the price could get a better parsing though; at least to convert -them to real decimals. Let's experiment a bit more:: - - >>> from decimal import Decimal # We'll use Decimal objects for storing the price and weight - - # We'll also use the same item, since in this case we'll be working with a different attribute - >>> item.set_attrib_adaptors('price', [ adaptors.Delist(), Decimal ]) - >>> item.attribute('price', product_rows[0].x('td[@class="prod_attrib"][5]/text()').re('(\d+)')) - >>> item.price - Decimal('300') - -In this case it wasn't necessary to use the extract adaptor, because applying -the ``re`` method over a selector already extracts the resulting content. - -Now, the price was quite easy, but the weight is a bit tricky, for the simple -fact that it has more than one weighing unit. In this case, we'll have to write -our own little adaptor that takes care of parsing a string and returning a -Decimal according to its unit. - -Something like this could work, although it's very primitive:: - - def parse_weight(string): - '''This adaptor receives a string and tries to parse it - as weight, converting it to grams and returning a Decimal object''' - - conversions = { - 'kg': Decimal('0.001'), - 'gr': Decimal('1'), - 'mg': Decimal('1000'), - } - - quantity = re.search('(\d+)', string) - if not quantity: - return Decimal('0') - quantity = Decimal(quantity.group(1)) - - for unit in conversions: - if unit in string: - quantity = quantity / conversions[unit] - break - return quantity - -And now some testing:: - - >>> parse_weight('4kg') - Decimal('4000') - >>> parse_weight('200 gr.') - Decimal('200') - >>> parse_weight('100 mg') - Decimal('0.1') - - >>> item.set_attrib_adaptors('weight', [ - adaptors.extract, - adaptors.Delist(), - parse_weight ]) - >>> item.attribute('weight', product_rows[0].x('td[@class="prod_attrib"][4]/text()').re('(\d+)')) - >>> item.weight - Decimal('2000') - -Ok, done! Let's now sum this up into a spider:: - - from decimal import Decimal - from scrapy.item import ScrapedItem - from scrapy.contrib_exp import adaptors - from scrapy.contrib.spiders import CrawlSpider, Rule - from scrapy.xpath.selector import HtmlXPathSelector - from scrapy.link.extractors import RegexLinkExtractor - - def parse_weight(string): - conversions = { - 'kg': Decimal('0.001'), - 'gr': Decimal('1'), - 'mg': Decimal('1000'), - } - - quantity = re.search('(\d+)', string) - if not quantity: - return Decimal('0') - quantity = Decimal(quantity.group(1)) - - for unit in conversions: - if unit in string: - quantity = quantity / conversions[unit] - break - return quantity - - class MySpider(CrawlSpider): - domain_name = 'scrapy.org' - start_urls = ['http://doc.scrapy.org/_static/items_adaptors-sample1.html'] - - rules = ( - Rule(RegexLinkExtractor(allow=(r'sample\d+\.html', )), 'parse_page'), - ) - - adaptors = { - 'manufacturer': [ adaptors.extract, adaptors.Unquote(), adaptors.Delist() ], - 'name': [ adaptors.extract, adaptors.Unquote(), adaptors.Delist() ], - 'description': [ adaptors.extract, adaptors.Unquote(), adaptors.Delist() ], - 'weight': [ adaptors.extract, adaptors.Delist(), parse_weight ], - 'price': [ adaptors.Delist(), Decimal ] - } - - def parse_page(self, response): - items = [] - rows = hxs.x('//tr[child::td[@class="prod_attrib"]]') - for product in rows: - item = ScrapedItem() - item.set_adaptors(self.adaptors) - - item.attribute('manufacturer', product.x('td[@class="prod_attrib"][1]/text()')) - item.attribute('name', product.x('td[@class="prod_attrib"][2]/text()')) - item.attribute('description', product.x('td[@class="prod_attrib"][3]/text()')) - item.attribute('weight', product.x('td[@class="prod_attrib"][4]/text()')) - item.attribute('price', product.x('td[@class="prod_attrib"][5]/text()').re('(\d+)')) - items.append(item) - - return items - - SPIDER = MySpider() - - -Basically this spider looks for the product rows in the page, creates an item -for each of them, attaches them some adaptors, and fills their attributes in. - -Scraping the sample page with this code would give us these items:: - - ScrapedItem({u"name": u"Bananas", u"manufacturer": u"Bill & Ted's farm", u"description": "Delicious fruit", u"weight": Decimal("2000"), u"price": Decimal("300")}) - ScrapedItem({u"name": u"Apple pie", u"manufacturer": u"Grandma's", u"description": "Grandma's best dish", u"weight": Decimal("250"), u"price": Decimal("200")}) - -There could be more parsing done here through adaptors, like parsing different -price currencies, or more advanced weight parsing (this one was very, very, -simple and buggy). Nevertheless, I hope that this was useful as an example use -of adaptors. - - -More complex adaptors -===================== - -Okay, what happens when you have adaptors that receive extra parameters in -order to modify their behaviour? For example, going back to money, what if you -have a function capable of converting from one currency to another one by -receiving three parameters: the value, its currency, and the currency to -convert to?. Well, don't worry, Scrapy can handle this situation and use this -adaptor too, and in fact, its something quite easy to do. - -Basically the difference between a regular adaptor and a "complex" one (I call -them complex just for the fact that the others are *very* simple), is that the -first one always receives and returns a single value, while the others may -receive more, passed by you to the item's ``attribute`` method. Whenever this -method (``attribute``) receives any extra parameters (apart from the value to -set, ``override``, and ``add``), it automatically passes them to any adaptor -that receives a parameter called ``adaptor_args`` in a dictionary. This -adaptor should read the dictionary and check if it has received any parameters -there. - -Notice that you should be careful with the naming of these parameters in order -to not generate confusion between adaptors, because every adaptor (that has the -``adaptor_args`` parameter, of course) will receive the same dict of keywords, -containing keywords that are destined for itself, and some that aren't. - -For example, it's not very wise to use a parameter called 'encoding' unless -you'd like all the adaptors to receive it; otherwise it'd be better to use the -adaptor's name as a prefix to the parameter, just to make it clearer. - -Let's now see the currency example, now written in code. - -First, we'll define a class called Currency and a dictionary, in order to store -some currencies and their exchange rate:: - - class Currency(object): - def __init__(self, from_dollars, to_dollars): - self._from_dollars = from_dollars - self._to_dollars = to_dollars - - def is_decimal(self, value): - if not isinstance(value, Decimal): - raise Exception('Value must be a Decimal object') - return True - - def from_dollars(self, value): - if self.is_decimal(value): - return value * self._from_dollars - - def to_dollars(self, value): - if self.is_decimal(value): - return value * self._to_dollars - - currencies = { - 'dollar': Currency(from_dollars=Decimal('1'), to_dollars=Decimal('1')), - 'euro': Currency(from_dollars=Decimal('0.7294'), to_dollars=Decimal('1.3710')), - 'yen': Currency(from_dollars=Decimal('90.82'), to_dollars=Decimal('0.0110')), - 'pound': Currency(from_dollars=Decimal('0.6547'), to_dollars=Decimal('1.5274')), - } - -Ok, now that we've got some exchange information, the only thing missing is the -adaptor that makes use of it:: - - def exchange_adaptor(value, adaptor_args): - currency_from = adaptor_args.get('currency_from', 'dollar') - currency_to = adaptor_args.get('currency_to', 'dollar') +But let's now talk a bit about adaptors (singularly), what are them, and how should they be implemented? +Adaptors are basically, any callable that receives a value, modifies it, and returns a new value (or more) so that the next +adaptor goes on with another adapting task (or not). +This is done this way to make the process of modifying information very customizable, and also to make adaptors reusable, +since they are intended to be small functions designed for simple purposes that can be applied in many different cases. +For example, you could make an adaptor for removing any tags in a text, like this:: - if currency_from in currencies and currency_to in currencies: - dollars = currencies[currency_from].to_dollars(value) - return currencies[currency_to].from_dollars(dollars) - else: - raise Exception('Unsupported currencies were specified') + >>> B_TAG_RE = re.compile(r'') + >>> def remove_b_tags(text): + >>> return B_TAG_RE.sub('', text) -And finally, testing it! :: +Then you could easily add this adaptor to a certain attribute's pipeline like this:: - >>> item = ScrapedItem() - >>> item.set_attrib_adaptors('price', [ Decimal, exchange_adaptor ]) - >>> item.attribute('price', '20.5', currency_from='euro', currency_to='dollar') - >>> item.price - Decimal('28.10550') + >>> item = MyItem() + >>> item.add_adaptor('text', remove_b_tags) + >>> item.attribute('text', u'some random text in bold and some random text in normal font') + >>> item.text + u'some random text in bold and some random text in normal font' +As you can see, this would make any value that you set to the item through the `attribute` method first pass through the +`remove_b_tags` adaptor, which would also replace any matching tag with an empty string. + +---- + +But anyway, let's now think of a bit more complicated (and useless) example: let's say you want to scrape a text, split it into single +letters, strip the vowels, turn the rest to capital letters, and join them again. +In this case, we could use three simple adaptors to process our data, plus a customized RobustScrapedItem for joining single +text attributes; let's see an example:: + + >>> # First of all, we define the item class we're going to use + >>> from string import ascii_letters + >>> from scrapy.contrib.item import RobustScrapedItem + >>> class MyItem(RobustScrapedItem): + >>> ATTRIBUTES = { + >>> 'text': basestring, + >>> } + + >>> def _add_single_attributes(self, attrname, attrtype, attributes): + >>> return ''.join(attributes) + + >>> # Now we'll write the needed adaptors + >>> def to_letters(text): + >>> return tuple(letter for letter in text) + + >>> def is_vowel(letter): + >>> if letter in ascii_letters and letter.lower() not in ('a', 'e', 'i', 'o', 'u'): + >>> return letter + + >>> def to_upper(letter): + >>> return letter.upper() + + >>> # Finally, we'll join all the pieces and see how it works + >>> item = MyItem() + >>> item.set_attrib_adaptors('text', [ + >>> to_letters, + >>> is_vowel, + >>> to_upper, + >>> ]) + +Let's now try with an example text to see what happens:: + + >>> item.attribute('text', 'pi', 'wind', add=True) + >>> item.text + 'PWND' + +Debugging +========= + +While you're coding spiders and adaptors, you usually need to know exactly what does Scrapy +do under the hood with the values you provide. +There's a setting called :setting:`ADAPTORS_DEBUG` for this purpose that makes Scrapy print +debugging messages each time an adaptors pipeline is run, specifying which attribute is being +adapted data for, the input/output values of each adaptor in the pipeline, and the input/output +of `_add_single_attributes` (in some cases). + +You can enable this setting as any other, either by adding it to your settings file, or by enabling +the environment variable `SCRAPY_ADAPTORS_DEBUG`. diff --git a/scrapy/trunk/docs/topics/items.rst b/scrapy/trunk/docs/topics/items.rst index 6a915fa4d..0cbf1b53b 100644 --- a/scrapy/trunk/docs/topics/items.rst +++ b/scrapy/trunk/docs/topics/items.rst @@ -4,9 +4,6 @@ Items ===== -.. module:: scrapy.item - :synopsis: Objects for storing scraped data - Quick overview ============== @@ -16,114 +13,130 @@ Quick overview ScrapedItems ============ +.. module:: scrapy.item + :synopsis: Objects for storing scraped data + .. class:: ScrapedItem Methods ------- -.. method:: ScrapedItem.__init__(data={}) +.. method:: ScrapedItem.__init__(data=None) - Instanciates a ``ScrapedItem`` object and sets an attribute and its value for each key in the given ``data`` dict. + :param data: A dictionary containing attributes and values to be set after instancing the item. -.. method:: ScrapedItem.attribute(self, attrname, value, override=False, add=False, **kwargs) - - Sets the item's ``attrname`` attribute with the given ``value`` filtering it through the attribute's adaptor pipeline (if any). - - ``attrname`` is a string containing the name of the attribute you're setting. - - ``value`` is the value you want to assign, which will be adapted by the corresponding adaptors for the given attribute (if any). - - ``override``, if True, makes this method avoid checking if there was a previous value and sets ``value`` no matter what. - - ``add``, if True, tries to concatenate the given ``value`` with the one already set in the item. This will work as long as - the old value is a list (in which case the new value will be appended, or the list will be extended if both are lists), - or as long as both values are strings (in which case ``add`` will be used as the delimiter, or default to '' if ``add=True``). - - ``kwargs`` - any extra parameters will be passed to any adaptor that receives an 'adaptor_args' parameter as a dictionary. - Check the Adaptors reference for more information. - -.. method:: ScrapedItem.set_adaptors(self, adaptors_dict) - - Receives a dict containing a list of adaptors for each desired attribute (key) and sets each of them as their adaptor pipeline. - -.. method:: ScrapedItem.set_attrib_adaptors(self, attrib, pipe) - - Sets the provided iterable (``pipe``) as the adaptor pipeline for the given attribute (``attrib``) - -.. method:: ScrapedItem.add_adaptor(self, attrib, adaptor, position=None) - - Adds an adaptor to an already existing (or not) pipeline. - - ``attr`` is the name of the attribute you're adding adaptors to. - - ``adaptor`` is a callable to be added to the pipeline. - - ``position`` is an integer representing the place where to add the adaptor. - If it's ``None``, the adaptor will be appended at the end of the pipeline. + Instanciates a ``ScrapedItem`` object and sets an attribute and its value for each key in the given ``data`` + dict (if any). + These items are the most basic items available, and the common interface from which any items should inherit. Examples -------- -Setting some basic attributes to a newly created item:: +Creating an item and setting some attributes:: >>> from scrapy.item import ScrapedItem - >>> person = ScrapedItem() - >>> person.attribute('name', 'John') - >>> person.attribute('age', 35) + >>> item = ScrapedItem() + >>> item.name = 'John' + >>> item.last_name = 'Smith' + >>> item.age = 23 + >>> item + ScrapedItem({'age': 23, 'last_name': 'Smith', 'name': 'John'}) + +Creating an item and setting its attributes inline:: + + >>> person = ScrapedItem({'name': 'John', 'age': 23, 'last_name': 'Smith'}) >>> person - ScrapedItem({'age': 35, 'name': 'John'}) + ScrapedItem({'age': 23, 'last_name': 'Smith', 'name': 'John'}) -We can also create an item and set its attributes by passing them inline using a dictionary, like:: +RobustScrapedItems +================== - >>> person = ScrapedItem({'name': 'John', 'age': 35}) - >>> person - ScrapedItem({'age': 35, 'name': 'John'}) +.. module:: scrapy.contrib.item + :synopsis: Objects for storing scraped data -Also, notice that making consecutive calls to the attribute method does *not* change its value, unless you use the `override` parameter:: +.. class:: RobustScrapedItem - >>> person = ScrapedItem() - >>> person.attribute('name', 'John') - >>> person - ScrapedItem({'name': 'John'}) + RobustScrapedItems are more complex items (compared to ScrapedItems) and have a few more features available, which + include: - >>> person.attribute('name', 'Charlie') - >>> person - ScrapedItem({'name': 'John'}) + * Attributes dictionary: items that inherit from RobustScrapedItem are defined with a dictionary of attributes in the class. + This allows the item to have much more logic at the moment of handling and setting attributes. The next features are + built on top of this one. - >>> person.attribute('name', 'Charlie', override=True) - >>> person - ScrapedItem({'name': 'Charlie'}) + * Adaptors: maybe the most important of the features that these items provide. The adaptors are a system designed for + filtering/modifying data before setting it to the item, that makes cleansing tasks *a lot* easier. -There's also an `add` parameter useful for concatenating lists or strings given a delimiter (or not):: + * Type checking: RobustScrapedItems come with a built-in type checking which assures you that no data of the wrong type will + get into the items without raising a warning. - >>> person = ScrapedItem() - >>> person.attribute('name', 'John') - >>> person - ScrapedItem({'name': 'John'}) + * Versioning: These items also provide versioning by making a unique hash for each item based on its attributes values. - # If add is True, '' is used as the default delimiter for joining strings - >>> person.attribute('name', 'Doe', add=True) - >>> person - ScrapedItem({'name': 'JohnDoe'}) + * ItemDeltas: You can subtract two RobustScrapedItems, which allows you to know the difference between a pair of items. + This difference is represented by a RobustItemDelta object. - # Otherwise, you can specify your own delimiter - >>> person.attribute('name', 'Smith', add=' ') - >>> person - ScrapedItem({'name': 'JohnDoe Smith'}) +Methods +------- - >>> person.attribute('children', ['Ken', 'Tom']) - >>> person - ScrapedItem({'name': 'JohnDoe Smith', 'children': ['Ken', 'Tom']}) +.. method:: RobustScrapedItem.__init__(data=None, adaptor_args=None) - # You can also append to lists... - >>> person.attribute('children', 'Billy', add=True) - >>> person - ScrapedItem({'name': 'JohnDoe Smith', 'children': ['Ken', 'Tom', 'Billy']}) + :param data: Idem as in ScrapedItems + :param adaptor_args: A dictionary of the like "attribute -> list of adaptors" for defining adaptors automatically after + instancing the item. - # And even extend them - >>> person.attribute('children', ['Dan', 'George'], add=True) - >>> person - ScrapedItem({'name': 'JohnDoe Smith', 'children': ['Ken', 'Tom', 'Billy', 'Dan', 'George']}) + Constructor of RobustScrapedItem objects. + +.. method:: RobustScrapedItem.attribute(self, attrname, value, override=False, add=False, ***kwargs) + + Sets the item's ``attrname`` attribute with the given ``value`` filtering it through the given attribute's adaptor + pipeline (if any). + + :param attrname: a string containing the name of the attribute you want to set. + + :param value: the value you want to assign, which will be adapted by the corresponding adaptors for the given attribute (if any). + + :param override: if True, makes this method avoid checking if there was a previous value and sets ``value`` no matter what. + + :param add: if True, tries to concatenate the given ``value`` with the one already set in the item. + For multivalued attributes, this will extend the list of already-set values, with the new ones. + For single valued attributes, the method _add_single_attributes (which is explained below) will be called. + + :param kwargs: any extra parameters will be passed in a dictionary to any adaptor that receives a parameter called 'adaptor_args'. + Check the :ref:`topics-adaptors` topic for more information. + +.. method:: RobustScrapedItem.set_adaptors(self, adaptors_dict) + + Receives a dict containing a list of adaptors for each desired attribute (key) and sets each of them as their adaptor pipeline. + +.. method:: RobustScrapedItem.set_attrib_adaptors(self, attrib, pipe) + + Sets the provided iterable (``pipe``) as the adaptor pipeline for the given attribute (``attrib``) + +.. method:: RobustScrapedItem.add_adaptor(self, attrib, adaptor, position=None) + + Adds an adaptor to an already existing (or not) pipeline. + + :param attr: the name of the attribute you're adding adaptors to. + + :param adaptor: a callable to be added to the pipeline. + + :param position: an integer representing the place where to add the adaptor. + If it's `None`, the adaptor will be appended at the end of the pipeline. + +Examples +-------- + +Creating a pretty basic item with a few attributes:: + + from scrapy.contrib.item import RobustScrapedItem + + class MyItem(RobustScrapedItem): + ATTRIBUTES = { + 'name': basestring, + 'size': basestring, + 'colours': [basestring], + } + +.. note:: + + More RobustScrapedItem examples are about to come. In the meantime, check the :ref:`topics-adaptors` topic to see a few of them. -Now, normally when we're scraping an HTML file, or almost any kind of file, information doesn't come to us exactly as we need it. We usually -have to make some adaptations here and there; and that's when the adaptors enter the game.