From 0227c53b13bb17bd442c591f0ee1ab4fb85dddb2 Mon Sep 17 00:00:00 2001 From: Ismael Carnales Date: Tue, 14 Jul 2009 15:55:52 -0300 Subject: [PATCH] reorganize newitem documentation --- docs/experimental/index.rst | 2 +- docs/experimental/newitem-adaptors.rst | 139 ++++++++++++++ docs/experimental/newitem-fields.rst | 150 --------------- docs/experimental/newitem.rst | 254 ++++++++++++++----------- 4 files changed, 280 insertions(+), 265 deletions(-) create mode 100644 docs/experimental/newitem-adaptors.rst delete mode 100644 docs/experimental/newitem-fields.rst diff --git a/docs/experimental/index.rst b/docs/experimental/index.rst index 12e5e697a..b0b046be5 100644 --- a/docs/experimental/index.rst +++ b/docs/experimental/index.rst @@ -20,5 +20,5 @@ it's not revised so frequently), or documentation which overlaps with existing :maxdepth: 1 newitem - newitem-fields + newitem-adaptors diff --git a/docs/experimental/newitem-adaptors.rst b/docs/experimental/newitem-adaptors.rst new file mode 100644 index 000000000..42e2a6123 --- /dev/null +++ b/docs/experimental/newitem-adaptors.rst @@ -0,0 +1,139 @@ +.. _topic-newitem-adaptors: + +============= +Item Adaptors +============= + +.. class:: scrapy.contrib_exp.newitem.adaptors.ItemAdaptor + +As you probably want to scrape the same kind of Items from many sources +(different websites, RSS feeds, etc.), Scrapy implements ItemAdaptors, they +allow you to adapt chunks of HTML or XML (selected using Selectors) to the +expected format of your Item fields. + +An ItemAdaptor acts like a wrapper of an Item, you define an ItemAdaptor class, +set the Item class to wrap and assign a set of functions (adaptor functions) to be called when you assign a value to a field. + +Here's an example of an ItemAdaptor for our previously created Item:: + + class NewsAdaptor(ItemAdaptor): + item_class = NewsItem + + url = adaptor(extract, remove_tags(), unquote(), strip) + headline = adaptor(extract, remove_tags(), unquote(), strip) + summary = adaptor(extract, remove_tags(), unquote(), strip) + content = adaptor(extract, remove_tags(), unquote(), strip) + +How do we use it? Let's see it in action in a Spider:: + + def parse_newspage(self, response): + xhs = HtmlXPathSelector(response) + i = NewsAdaptor(response) + + i.url = response.url + i.headline = xhs.x('//h1[@class="headline"]') + i.summary = xhs.x('//div[@class="summary"]') + i.content = xhs.x('//div[@id="body"]') + # published attribute is intentionally omitted, see below for site-specific adaptors + return [i] + +What happens underneath? + +When we assign a value to a ItemAdaptor field it passes for the chain of +functions defined previously in it's class, in this case, the value gets +extracted (note that we assign directly the value obtained from the Selector), +then tags will be removed, then the result will be unquoted, stripped and +finally assigned to the Item Field. + +This final assignment is done in an internal instance of the Item on the +ItemAdaptor, that's why we can return an ItemAdaptor instead of an Item and +Scrapy will know how to extract the item from it. + +An Item can have as many ItemAdaptors as you want. It generally depends on how +many sources and formats are you scraping from. + +ItemAdaptor inheritance +======================= + +As we said before you generally want an ItemAdaptor for each different source of +data and maybe some for specific sites, inheritance make this really easy, let's +see an example of adapting HTML and XML:: + + class NewsAdaptor(ItemAdaptor): + item_class = NewsItem + + + class HtmlNewsAdaptor(NewsAdaptor): + url = adaptor(extract, remove_tags(), unquote(), strip) + headline = adaptor(extract, remove_tags(), unquote(), strip) + summary = adaptor(extract, remove_tags(), unquote(), strip) + content = adaptor(extract, remove_tags(), unquote(), strip) + published = adaptor(extract, remove_tags(), unquote(), strip) + + + class XmlNewsAdaptor(HtmlNewsAdaptor): + url = adaptor(extract, remove_root, strip) + headline = adaptor(extract, remove_root, strip) + summary = adaptor(extract, remove_root, strip) + content = adaptor(extract, remove_root, strip) + published = adaptor(extract, remove_root, strip) + + +Site specific ItemAdaptors +========================== + +For the moment we have covered adapting information from different sources, but +other common case is adapting information for specific sites, think for example +in our published field, it keeps the publication date of the news article. + +As sites offer this information in different formats, we will have to make +custom adaptors for it, let's see an example using our Item published field:: + + class SpecificSiteNewsAdaptor(HtmlNewsAdaptor): + published = adaptor(HtmlNewsAdaptor.published, to_date('%d.%m.%Y')) + + +The ``to_date`` adaptor function converts a string with the format specified in +its parameter to one in 'YYYY-mm-dd' format (the one that DateField expects). + +And in this example we're appending it to the of the chain of adaptor functions +of published. + +Note that ``SpecificSiteNewsAdaptor`` will inherit the field adaptations from +``HtmlNewsAdaptor``. + +Let's see it in action:: + + def parse_newspage(self, response): + xhs = HtmlXPathSelector(response) + i = SpecificSiteNewsAdaptor(response) + + i.url = response.url + i.headline = xhs.x('//h1[@class="headline"]') + i.summary = xhs.x('//div[@class="summary"]') + i.content = xhs.x('//div[@id="body"]') + i.published = xhs.x('//h1[@class="date"]').re('\d{2}\.\d{2}\.\d{4}') + return [i] + +ItemAdaptor default_adaptor +=========================== + +If you look closely at the code for our ItemAdaptors you can see that we're +using the same set of adaptation functions in every field. + +It is common for ItemAdaptors to have a basic set of adaptor functions that +will be applied to almost every Field in the Item. To avoid repeating the same +code, ItemAdaptor implements the ``default_adaptor`` shortcut. + +``default_adaptor`` (if set) will be called when assigning a value for an Item +Field that has no adaptor, so the process for determining what value gets +assigned to an item when you assign a value to an ItemAdaptor field is as +follows: + +1. If there's an adaptor function for this field its called before assigning + the value to the item. +2. If no adaptor function if set and default_adaptor is, the value passes for + ``default_adaptor`` before being assigned. +3. If no adaptor is defined for that field and no ``default_adaptor`` is set, + the value is assigned directly. + diff --git a/docs/experimental/newitem-fields.rst b/docs/experimental/newitem-fields.rst deleted file mode 100644 index a6508708b..000000000 --- a/docs/experimental/newitem-fields.rst +++ /dev/null @@ -1,150 +0,0 @@ -.. _ref-newitem-fields: - -===================== -Item Fields Reference -===================== - -Item Fields can be used to specify the type of each Item attribute and its -default value. For example:: - - from scrapy.contrib_exp.newitem import Item, fields - - class NewsItem(Item): - content = fields.TextField() - author = fields.TextField(default=u'Myself') - published = fields.DateField() - views = fields.IntegerField(default=0) - -Fields which contain a default value will always return that value when not -set, while fields which don't contain a default value will always return -``None`` when not set:: - - >>> it = NewsItem() - >>> it.content is None - True - >>> it.author - u'Myself' - >>> it.published is None - True - >>> it.views - 0 - -All field classes are subclasses of the :class:`BaseField` class which you can -also subclass to create your own custom fields. But you can also subclass a -more specific field class, say :class:`DecimalField`, to implement a -``PriceField``, for example. - -.. module:: scrapy.contrib_exp.newitem.fields - -BaseField class -=============== - -.. class:: BaseField(default=None) - - The base class for all fields. It only provides code for handling default - values, not any particular type. It cannot be used directly either, as its - :meth:`BaseField.to_python` method is not implemented. - - The ``default`` argument (if given) must be of the type expected by this - field, or any type that is accepted by the :meth:``BaseField.to_python`` - method of this field. - - For example:: - - class NewsItem(Item): - content = fields.TextField() # correct, no default value - author = fields.TextField(default=u'Myself") # correct, with default value - published = fields.DateField(default=23) # wrong default type (will raise TypeError) - - .. method:: to_python(value) - - Convert the input value to the type expected by this field and return - it. - - For example, :class:`IntegerField` would convert ``'1'`` to ``1``, while - :class:`DecimalField` would convert ``'1'`` to ``Decimal('1')`` and so - on. - - This method is not implemented in the :class:`BaseField` class, so it - must always be implemented in all its subclasses, in order to be usable. - - This method should raise ``TypeError`` if the input type is not - supported, and ``ValueError`` if the input type is support but its value - is not appropriate (for example, an integer outside a given range). - - This method must always return object of the expected field type. - - .. method:: get_default() - - Return the default value for this field, or ``None`` if the field - doesn't specify any. - - -Available built-in Fields -========================= - -TextField ---------- - -.. class:: TextField - - A unicode text. - -IntegerField ------------- - -.. class:: IntegerField - - An integer. - -DecimalField ------------- - -.. class:: DecimalField - - A fixed-precision decimal number, represented in Python by a `Decimal`_ - instance. - -.. _Decimal: http://docs.python.org/library/decimal.html#decimal.Decimal - -FloatField ----------- - -.. class:: FloatField - - A floating-point number represented in Python by a ``float`` instance. - -BooleanField ------------- - -.. class:: BooleanField - - A boolean (true/false) field. - -DateTimeField -------------- - -.. class:: DateTimeField - - A date with time, represented in Python by a `datetime.datetime`_ instance. - -.. _datetime.datetime: http://docs.python.org/library/datetime.html#datetime.datetime - -DateField ---------- - -.. class:: DateField - - A date, represented in Python by a `datetime.date`_ instance. - -.. _datetime.date: http://docs.python.org/library/datetime.html#datetime.date - -TimeField ---------- - -.. class:: TimeField - - A time, represented in Python by a `datetime.time`_ instance. - -.. _datetime.time: http://docs.python.org/library/datetime.html#datetime.time - diff --git a/docs/experimental/newitem.rst b/docs/experimental/newitem.rst index 5796b67c0..7341db6e3 100644 --- a/docs/experimental/newitem.rst +++ b/docs/experimental/newitem.rst @@ -57,147 +57,173 @@ converted to the expected Python types depending of their class:: Each field accepts a ``default`` argument, that sets the default value of the field. -You can see the built-in field types in the :ref:`ref-newitem-fields`. - Using this may seen complicated at first, but gives you much power over scraped data, like assigning defaults for fields that are not present in some pages, :ref:`topic-newitem-adaptors`, etc. -.. _topic-newitem-adaptors: +.. _ref-newitem-fields: -============= -Item Adaptors +=========== +Item Fields +=========== + +.. module:: scrapy.contrib_exp.newitem.fields + +Field options ============= -.. class:: scrapy.contrib_exp.newitem.adaptors.ItemAdaptor +Every ``Field`` class constructor accepts these arguments. -As you probably want to scrape the same kind of Items from many sources -(different websites, RSS feeds, etc.), Scrapy implements ItemAdaptors, they -allow you to adapt chunks of HTML or XML (selected using Selectors) to the -expected format of your Item fields. +``default`` +----------- -An ItemAdaptor acts like a wrapper of an Item, you define an ItemAdaptor class, -set the Item class to wrap and assign a set of functions (adaptor functions) to be called when you assign a value to a field. +The default value for the field. -Here's an example of an ItemAdaptor for our previously created Item:: +Fields which contain a default value will always return that value when not +set, while fields which don't contain a default value will always return +``None`` when not set:: - class NewsAdaptor(ItemAdaptor): - item_class = NewsItem + from scrapy.contrib_exp.newitem import Item, fields - url = adaptor(extract, remove_tags(), unquote(), strip) - headline = adaptor(extract, remove_tags(), unquote(), strip) - summary = adaptor(extract, remove_tags(), unquote(), strip) - content = adaptor(extract, remove_tags(), unquote(), strip) + class NewsItem(Item): + content = fields.TextField() + author = fields.TextField(default=u'Myself') + published = fields.DateField() + views = fields.IntegerField(default=0) -How do we use it? Let's see it in action in a Spider:: + >>> it = NewsItem() + >>> it.content is None + True + >>> it.author + u'Myself' + >>> it.published is None + True + >>> it.views + 0 - def parse_newspage(self, response): - xhs = HtmlXPathSelector(response) - i = NewsAdaptor(response) +Field types +=========== - i.url = response.url - i.headline = xhs.x('//h1[@class="headline"]') - i.summary = xhs.x('//div[@class="summary"]') - i.content = xhs.x('//div[@id="body"]') - # published attribute is intentionally omitted, see below for site-specific adaptors - return [i] +These are the available built-in ``Field`` types. See +:ref:`newitem-custom-fields` for info on creating your own field types. -What happens underneath? +TextField +--------- -When we assign a value to a ItemAdaptor field it passes for the chain of -functions defined previously in it's class, in this case, the value gets -extracted (note that we assign directly the value obtained from the Selector), -then tags will be removed, then the result will be unquoted, stripped and -finally assigned to the Item Field. +.. class:: TextField -This final assignment is done in an internal instance of the Item on the -ItemAdaptor, that's why we can return an ItemAdaptor instead of an Item and -Scrapy will know how to extract the item from it. + A unicode text. -An Item can have as many ItemAdaptors as you want. It generally depends on how -many sources and formats are you scraping from. +IntegerField +------------ -ItemAdaptor inheritance -======================= +.. class:: IntegerField -As we said before you generally want an ItemAdaptor for each different source of -data and maybe some for specific sites, inheritance make this really easy, let's -see an example of adapting HTML and XML:: + An integer. - class NewsAdaptor(ItemAdaptor): - item_class = NewsItem +DecimalField +------------ + +.. class:: DecimalField + + A fixed-precision decimal number, represented in Python by a `Decimal`_ + instance. + +.. _Decimal: http://docs.python.org/library/decimal.html#decimal.Decimal + +FloatField +---------- + +.. class:: FloatField + + A floating-point number represented in Python by a ``float`` instance. + +BooleanField +------------ + +.. class:: BooleanField + + A boolean (true/false) field. + +DateTimeField +------------- + +.. class:: DateTimeField + + A date with time, represented in Python by a `datetime.datetime`_ instance. + +.. _datetime.datetime: http://docs.python.org/library/datetime.html#datetime.datetime + +DateField +--------- + +.. class:: DateField + + A date, represented in Python by a `datetime.date`_ instance. + +.. _datetime.date: http://docs.python.org/library/datetime.html#datetime.date + +TimeField +--------- + +.. class:: TimeField + + A time, represented in Python by a `datetime.time`_ instance. + +.. _datetime.time: http://docs.python.org/library/datetime.html#datetime.time - class HtmlNewsAdaptor(NewsAdaptor): - url = adaptor(extract, remove_tags(), unquote(), strip) - headline = adaptor(extract, remove_tags(), unquote(), strip) - summary = adaptor(extract, remove_tags(), unquote(), strip) - content = adaptor(extract, remove_tags(), unquote(), strip) - published = adaptor(extract, remove_tags(), unquote(), strip) +.. _newitem-custom-fields: +Creating custom fields +====================== + +All field classes are subclasses of the :class:`BaseField` class (see below) +which you can also subclass to create your own custom fields. + +You can also subclass a more specific field class, say :class:`DecimalField`, +to implement a ``PriceField``, for example. + +BaseField class +--------------- + +.. class:: BaseField(default=None) + + The base class for all fields. It only provides code for handling default + values, not any particular type. It cannot be used directly either, as its + :meth:`BaseField.to_python` method is not implemented. + + The ``default`` argument (if given) must be of the type expected by this + field, or any type that is accepted by the :meth:``BaseField.to_python`` + method of this field. + + For example:: + + class NewsItem(Item): + content = fields.TextField() # correct, no default value + author = fields.TextField(default=u'Myself") # correct, with default value + published = fields.DateField(default=23) # wrong default type (will raise TypeError) + + .. method:: to_python(value) + + Convert the input value to the type expected by this field and return + it. - class XmlNewsAdaptor(HtmlNewsAdaptor): - url = adaptor(extract, remove_root, strip) - headline = adaptor(extract, remove_root, strip) - summary = adaptor(extract, remove_root, strip) - content = adaptor(extract, remove_root, strip) - published = adaptor(extract, remove_root, strip) + For example, :class:`IntegerField` would convert ``'1'`` to ``1``, while + :class:`DecimalField` would convert ``'1'`` to ``Decimal('1')`` and so + on. + + This method is not implemented in the :class:`BaseField` class, so it + must always be implemented in all its subclasses, in order to be usable. + This method should raise ``TypeError`` if the input type is not + supported, and ``ValueError`` if the input type is support but its value + is not appropriate (for example, an integer outside a given range). -Site specific ItemAdaptors -========================== + This method must always return object of the expected field type. + + .. method:: get_default() -For the moment we have covered adapting information from different sources, but -other common case is adapting information for specific sites, think for example -in our published field, it keeps the publication date of the news article. + Return the default value for this field, or ``None`` if the field + doesn't specify any. -As sites offer this information in different formats, we will have to make -custom adaptors for it, let's see an example using our Item published field:: - - class SpecificSiteNewsAdaptor(HtmlNewsAdaptor): - published = adaptor(HtmlNewsAdaptor.published, to_date('%d.%m.%Y')) - - -The ``to_date`` adaptor function converts a string with the format specified in -its parameter to one in 'YYYY-mm-dd' format (the one that DateField expects). - -And in this example we're appending it to the of the chain of adaptor functions -of published. - -Note that ``SpecificSiteNewsAdaptor`` will inherit the field adaptations from -``HtmlNewsAdaptor``. - -Let's see it in action:: - - def parse_newspage(self, response): - xhs = HtmlXPathSelector(response) - i = SpecificSiteNewsAdaptor(response) - - i.url = response.url - i.headline = xhs.x('//h1[@class="headline"]') - i.summary = xhs.x('//div[@class="summary"]') - i.content = xhs.x('//div[@id="body"]') - i.published = xhs.x('//h1[@class="date"]').re('\d{2}\.\d{2}\.\d{4}') - return [i] - -ItemAdaptor default_adaptor -=========================== - -If you look closely at the code for our ItemAdaptors you can see that we're -using the same set of adaptation functions in every field. - -It is common for ItemAdaptors to have a basic set of adaptor functions that -will be applied to almost every Field in the Item. To avoid repeating the same -code, ItemAdaptor implements the ``default_adaptor`` shortcut. - -``default_adaptor`` (if set) will be called when assigning a value for an Item -Field that has no adaptor, so the process for determining what value gets -assigned to an item when you assign a value to an ItemAdaptor field is as -follows: - -1. If there's an adaptor function for this field its called before assigning - the value to the item. -2. If no adaptor function if set and default_adaptor is, the value passes for - ``default_adaptor`` before being assigned. -3. If no adaptor is defined for that field and no ``default_adaptor`` is set, - the value is assigned directly.