diff --git a/scrapy/trunk/docs/ref/exceptions.rst b/scrapy/trunk/docs/ref/exceptions.rst index 31ffee63c..91d4b3230 100644 --- a/scrapy/trunk/docs/ref/exceptions.rst +++ b/scrapy/trunk/docs/ref/exceptions.rst @@ -20,7 +20,7 @@ DropItem -------- The exception that must be raised by item pipeline stages to stop processing an -Item. For more information see :topic:`itempipeline`. +Item. For more information see :topic:`item-pipeline`. .. exception:: NotConfigured diff --git a/scrapy/trunk/docs/ref/signals.rst b/scrapy/trunk/docs/ref/signals.rst index 56c44b9cc..c8aee5739 100644 --- a/scrapy/trunk/docs/ref/signals.rst +++ b/scrapy/trunk/docs/ref/signals.rst @@ -141,7 +141,7 @@ Arguments: * ``response`` - the response from which the item was scraped Sent when the engine receives a new scraped item from the spider, and right -before the item is sent to the :topic:`itempipeline`. +before the item is sent to the :topic:`item-pipeline`. .. signal:: item_passed @@ -155,7 +155,7 @@ Arguments: * ``pipe_output`` - the output of the item pipeline. Typically, this points to the same ``item`` object, unless some pipeline stage created a new item. -Sent after an item has passed al the :topic:`itempipeline` stages without +Sent after an item has passed al the :topic:`item-pipeline` stages without being dropped. .. signal:: item_dropped @@ -169,7 +169,7 @@ Arguments: * ``response`` - the response from which the item was scraped * ``exception`` - the exception that caused the item to be dropped (which must inherit from :exception:`DropItem`) -Sent after an item has dropped from the :topic:`itempipeline` when some stage +Sent after an item has dropped from the :topic:`item-pipeline` when some stage raised a :exception:`DropItem` exception. diff --git a/scrapy/trunk/docs/topics/index.rst b/scrapy/trunk/docs/topics/index.rst index 3bafbc426..c6a3c37dd 100644 --- a/scrapy/trunk/docs/topics/index.rst +++ b/scrapy/trunk/docs/topics/index.rst @@ -4,7 +4,7 @@ Topics .. toctree:: :maxdepth: 1 - itempipeline + item-pipeline downloader-middleware selectors settings diff --git a/scrapy/trunk/docs/topics/item-pipeline.rst b/scrapy/trunk/docs/topics/item-pipeline.rst new file mode 100644 index 000000000..2745607d7 --- /dev/null +++ b/scrapy/trunk/docs/topics/item-pipeline.rst @@ -0,0 +1,68 @@ +============= +Item Pipeline +============= + +After an item has been scraped by a spider it is sent to the Item Pipeline +which process it through several components that are executed sequentially. + +Item pipeline are usually implemented on each project. Typical usage for item +pipelines are: + + * HTML cleansing + * validation + * persistence (storing the scraped item) + + +Writing your own item pipeline +============================== + +Writing your own item pipeline is easy. Each item pipeline component is a +single Python class that must define the following method: + +.. method:: process_item(request, spider) + +``request`` is a Request object +``spider`` is a BaseSpider object + +This method is called for every item pipeline component and must either return +a ScrapedItem (or any descendant class) object or raise a :exception:`DropItem` +exception. Dropped items are no longer processed by further pipeline +components. + + +The class can also define the followin optional methods: + +.. method:: open_domain(domain) + +``domain`` is a string which identifies the spider opened + +This method is called when a spider is opened for crawling. + +.. method:: close_domain(domain) + +``domain`` is a string which identifies the spider closed + +This method is called when a spider is closed. + + +Item pipeline example +===================== + +Let's take a look at following hypotetic pipeline that adjusts the ``price`` +attribute for those items that do not include VAT (``price_excludes_vat`` +attribute), and drops those items which don't contain a price:: + + from scrapy.core.exceptions import DropItem + + class PricePipeline(object): + + vat_factor = 1.15 + + def process_item(self, domain, response, item): + if item.price: + if item.price_excludes_vat: + item.price = item.price * self.vat_factor + return item + else: + raise DropItem("Missing price in %s" % item) + diff --git a/scrapy/trunk/docs/topics/itempipeline.rst b/scrapy/trunk/docs/topics/itempipeline.rst deleted file mode 100644 index 92cc60da0..000000000 --- a/scrapy/trunk/docs/topics/itempipeline.rst +++ /dev/null @@ -1,48 +0,0 @@ -============= -Item Pipeline -============= - -After an item has been scraped by a spider it is sent to the Item Pipeline -which process it through several stages that are executed sequentially. - -Item pipeline are usually implemented on each project. Typical usage for item -pipelines are: - - * HTML cleansing - * validation - * persistence (storing the scraped item) - -To implement an item pipeline you need to implement a class which contains a -``process_item`` method. That method must either return a ScrapedItem (or any -descendant class) object or raise a :exception:`DropItem` exception. Dropped -items are no longer processed by further pipeline stages. - -Let's take a look at following hypotetic pipeline that adjusts the ``price`` -attribute for those items that do not include VAT (``price_excludes_vat`` -attribute), and drops those items which don't contain a price:: - - from scrapy.core.exceptions import DropItem - - class PricePipeline(object): - - vat_factor = 1.15 - - def process_item(self, domain, response, item): - if item.price: - if item.price_excludes_vat: - item.price = item.price * self.vat_factor - return item - else: - raise DropItem("Missing price in %s" % item) - - def open_domain(self, domain): - pass - - def close_domains(self, domain): - pass - - -You may also noticed the two other methods that a item pipeline class can -contain: ``open_domain`` which is called when the domain for that spider is -open, and ``close_domain`` which is called when it's closed. -