From 817dbc6cbd04d1ee8644ccc22d1b109afcf5a892 Mon Sep 17 00:00:00 2001 From: Mikhail Korobov Date: Thu, 19 Mar 2015 05:16:14 +0500 Subject: [PATCH] DOC mention dicts in documentation; explain better what are Items for --- docs/topics/architecture.rst | 4 ++-- docs/topics/exporters.rst | 13 +++++++------ docs/topics/images.rst | 13 +++++++++---- docs/topics/item-pipeline.rst | 15 ++++++++------- docs/topics/items.rst | 17 ++++++++++++----- docs/topics/practices.rst | 7 +++---- docs/topics/signals.rst | 4 ++-- docs/topics/spider-middleware.rst | 9 +++++---- 8 files changed, 48 insertions(+), 34 deletions(-) diff --git a/docs/topics/architecture.rst b/docs/topics/architecture.rst index 80ccd42dd..55df1cc98 100644 --- a/docs/topics/architecture.rst +++ b/docs/topics/architecture.rst @@ -102,10 +102,10 @@ this: 6. The Engine receives the Response from the Downloader and sends it to the Spider for processing, passing through the Spider Middleware (input direction). -7. The Spider processes the Response and returns scraped Items and new Requests +7. The Spider processes the Response and returns scraped items and new Requests (to follow) to the Engine. -8. The Engine sends scraped Items (returned by the Spider) to the Item Pipeline +8. The Engine sends scraped items (returned by the Spider) to the Item Pipeline and Requests (returned by spider) to the Scheduler 9. The process repeats (from step 2) until there are no more requests from the diff --git a/docs/topics/exporters.rst b/docs/topics/exporters.rst index 43931544f..7455b28da 100644 --- a/docs/topics/exporters.rst +++ b/docs/topics/exporters.rst @@ -7,7 +7,7 @@ Item Exporters .. module:: scrapy.contrib.exporter :synopsis: Item Exporters -Once you have scraped your Items, you often want to persist or export those +Once you have scraped your items, you often want to persist or export those items, to use the data in some other application. That is, after all, the whole purpose of the scraping process. @@ -90,9 +90,9 @@ described next. 1. Declaring a serializer in the field -------------------------------------- -You can declare a serializer in the :ref:`field metadata -`. The serializer must be a callable which receives a -value and returns its serialized form. +If you use :class:`~.Item` you can declare a serializer in the +:ref:`field metadata `. The serializer must be +a callable which receives a value and returns its serialized form. Example:: @@ -167,8 +167,9 @@ BaseItemExporter value unchanged except for ``unicode`` values which are encoded to ``str`` using the encoding declared in the :attr:`encoding` attribute. - :param field: the field being serialized - :type field: :class:`~scrapy.item.Field` object + :param field: the field being serialized. If a raw dict is being + exported (not :class:`~.Item`) *field* value is an empty dict. + :type field: :class:`~scrapy.item.Field` object or an empty dict :param name: the name of the field being serialized :type name: str diff --git a/docs/topics/images.rst b/docs/topics/images.rst index 91e59d2a9..2cbff2e84 100644 --- a/docs/topics/images.rst +++ b/docs/topics/images.rst @@ -63,9 +63,14 @@ this: Usage example ============= -In order to use the image pipeline you just need to :ref:`enable it -` and define an item with the ``image_urls`` and -``images`` fields:: +In order to use the image pipeline first +:ref:`enable it `. + +Then, if a spider returns a dict with 'image_urls' key, +the pipeline will put the results under 'images' key. + +If you prefer to use :class:`~.Item` then define a custom +item with the ``image_urls`` and ``images`` fields:: import scrapy @@ -74,7 +79,7 @@ In order to use the image pipeline you just need to :ref:`enable it # ... other item fields ... image_urls = scrapy.Field() images = scrapy.Field() - + If you need something more complex and want to override the custom images pipeline behaviour, see :ref:`topics-images-override`. diff --git a/docs/topics/item-pipeline.rst b/docs/topics/item-pipeline.rst index 7b66753b8..973c77516 100644 --- a/docs/topics/item-pipeline.rst +++ b/docs/topics/item-pipeline.rst @@ -8,8 +8,8 @@ After an item has been scraped by a spider, it is sent to the Item Pipeline which process it through several components that are executed sequentially. Each item pipeline component (sometimes referred as just "Item Pipeline") is a -Python class that implements a simple method. They receive an Item and perform -an action over it, also deciding if the Item should continue through the +Python class that implements a simple method. They receive an item and perform +an action over it, also deciding if the item should continue through the pipeline or be dropped and no longer processed. Typical use for item pipelines are: @@ -28,12 +28,12 @@ Each item pipeline component is a Python class that must implement the following .. method:: process_item(self, item, spider) This method is called for every item pipeline component and must either return - a :class:`~scrapy.item.Item` (or any descendant class) object or raise a - :exc:`~scrapy.exceptions.DropItem` exception. Dropped items are no longer + a dict with data, :class:`~scrapy.item.Item` (or any descendant class) object + or raise a :exc:`~scrapy.exceptions.DropItem` exception. Dropped items are no longer processed by further pipeline components. :param item: the item scraped - :type item: :class:`~scrapy.item.Item` object + :type item: :class:`~scrapy.item.Item` object or a dict :param spider: the spider which scraped the item :type spider: :class:`~scrapy.spider.Spider` object @@ -135,6 +135,8 @@ method and how to clean up the resources properly. import pymongo class MongoPipeline(object): + + collection_name = 'scrapy_items' def __init__(self, mongo_uri, mongo_db): self.mongo_uri = mongo_uri @@ -155,8 +157,7 @@ method and how to clean up the resources properly. self.client.close() def process_item(self, item, spider): - collection_name = item.__class__.__name__ - self.db[collection_name].insert(dict(item)) + self.db[self.collection_name].insert(dict(item)) return item .. _MongoDB: http://www.mongodb.org/ diff --git a/docs/topics/items.rst b/docs/topics/items.rst index ee604a7f1..3fda20494 100644 --- a/docs/topics/items.rst +++ b/docs/topics/items.rst @@ -8,12 +8,21 @@ Items :synopsis: Item and Field classes The main goal in scraping is to extract structured data from unstructured -sources, typically, web pages. Scrapy provides the :class:`Item` class for this -purpose. +sources, typically, web pages. Scrapy spiders can return the extracted data +as Python dicts. While convenient and familiar, Python dicts lack structure: +it is easy to make a typo in a field name or return inconsistent data, +especially in a larger project with many spiders. +To define common output data format Scrapy provides the :class:`Item` class. :class:`Item` objects are simple containers used to collect the scraped data. They provide a `dictionary-like`_ API with a convenient syntax for declaring -their available fields. +their available fields. + +Various Scrapy components use extra information provided by Items: +exporters look at declared fields to figure out columns to export, +serialization can be customized using Item fields metadata, :mod:`trackref` +tracks Item instances to help finding memory leaks +(see :ref:`topics-leaks-trackrefs`_), etc. .. _dictionary-like: http://docs.python.org/library/stdtypes.html#dict @@ -64,8 +73,6 @@ It's important to note that the :class:`Field` objects used to declare the item do not stay assigned as class attributes. Instead, they can be accessed through the :attr:`Item.fields` attribute. -And that's all you need to know about declaring items. - Working with Items ================== diff --git a/docs/topics/practices.rst b/docs/topics/practices.rst index 13dde52a3..9e65c07be 100644 --- a/docs/topics/practices.rst +++ b/docs/topics/practices.rst @@ -190,11 +190,10 @@ Dynamic Creation of Item Classes ================================ For applications in which the structure of item class is to be determined by -user input, or other changing conditions, you can dynamically create item -classes instead of manually coding them. - -:: +user input or other changing conditions you can return regular Python +dicts from spiders. +Another option is to dynamically create Item classes:: from scrapy.item import DictItem, Field diff --git a/docs/topics/signals.rst b/docs/topics/signals.rst index 405b131ed..85cf43c76 100644 --- a/docs/topics/signals.rst +++ b/docs/topics/signals.rst @@ -71,7 +71,7 @@ item_scraped This signal supports returning deferreds from their handlers. :param item: the item scraped - :type item: :class:`~scrapy.item.Item` object + :type item: dict or :class:`~scrapy.item.Item` object :param spider: the spider which scraped the item :type spider: :class:`~scrapy.spider.Spider` object @@ -91,7 +91,7 @@ item_dropped This signal supports returning deferreds from their handlers. :param item: the item dropped from the :ref:`topics-item-pipeline` - :type item: :class:`~scrapy.item.Item` object + :type item: dict or :class:`~scrapy.item.Item` object :param spider: the spider which scraped the item :type spider: :class:`~scrapy.spider.Spider` object diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst index 79cfbcf5d..0104ae552 100644 --- a/docs/topics/spider-middleware.rst +++ b/docs/topics/spider-middleware.rst @@ -90,15 +90,16 @@ following methods: it has processed the response. :meth:`process_spider_output` must return an iterable of - :class:`~scrapy.http.Request` or :class:`~scrapy.item.Item` objects. + :class:`~scrapy.http.Request`, dict or :class:`~scrapy.item.Item` + objects. :param response: the response which generated this output from the spider :type response: :class:`~scrapy.http.Response` object :param result: the result returned by the spider - :type result: an iterable of :class:`~scrapy.http.Request` or - :class:`~scrapy.item.Item` objects + :type result: an iterable of :class:`~scrapy.http.Request`, dict + or :class:`~scrapy.item.Item` objects :param spider: the spider whose result is being processed :type spider: :class:`~scrapy.spider.Spider` object @@ -110,7 +111,7 @@ following methods: method (from other spider middleware) raises an exception. :meth:`process_spider_exception` should return either ``None`` or an - iterable of :class:`~scrapy.http.Response` or + iterable of :class:`~scrapy.http.Response`, dict or :class:`~scrapy.item.Item` objects. If it returns ``None``, Scrapy will continue processing this exception,