provide documentation for nested loaders

This commit is contained in:
Daniel Collins 2015-08-29 14:23:25 -07:00 committed by Daniel Collins
parent 425e35ee90
commit 88c92cb68b
1 changed files with 54 additions and 0 deletions

View File

@ -432,6 +432,14 @@ ItemLoader objects
<topics-loaders-processors>` to get the final value to assign to each
item field.
.. method:: nested_loader(xpath=selector, css=selector)
Create a nested loader with either an xpath selector or css selector.
The supplied selector is applied relative to selector associated
with this :class:`ItemLoader`. The nested loader shares the :class:`Item`
with the parent :class:`ItemLoader` so calls to :meth:`add_xpath`,
:meth:`add_value`, :meth:`replace_value`, etc. will behave as expected.
.. method:: get_collected_values(field_name)
Return the collected values for the given field.
@ -490,6 +498,52 @@ ItemLoader objects
:attr:`default_selector_class`. This attribute is meant to be
read-only.
.. _topics-loaders-nested:
Nested Loaders
==============
When parsing related values from a subsection of a document, it can be
useful to create nested loaders. Imagine you're extracting details from
a footer of a page that looks something like:
Example::
<footer>
<a class="social" href="http://facebook.com/whatever">Like Us</a>
<a class="social" href="http://twitter.com/whatever">Follow Us</a>
<a class="email" href="mailto:whatever@example.com">Email Us</a>
</footer>
Without nested loaders, you need to specify the full xpath (or css) for each value
that you wish to extract.
Example::
loader = ItemLoader(item=Item())
# load stuff not in the footer
loader.add_xpath('social', '//footer/a[@class = "social"]/@href')
loader.add_xpath('email', '//footer/a[@class = "email"]/@href')
loader.load_item()
Instead, you can create a nested loader with the footer selector and add values
relative to the footer. The functionality is the same but you avoid repeating
the footer selector.
Example::
loader = ItemLoader(item=Item())
# load stuff not in the footer
footer_loader = loader.nested_loader(xpath='//footer')
footer_loader.add_xpath('social', 'a[@class = "social"]/@href')
footer_loader.add_xpath('email', 'a[@class = "email"]/@href')
# no need to call footer_loader.load_item()
loader.load_item()
You can nest loaders arbitrarilly and they work with either xpath or css selectors.
As a general guideline, use nested loaders when they make your code simpler but do
not go overboard with nesting or your parser can become difficult to read.
.. _topics-loaders-extending:
Reusing and extending Item Loaders