From 1bbe7991dc08c08f539775e1ec00a3f36f0c4dc1 Mon Sep 17 00:00:00 2001 From: Ismael Carnales Date: Tue, 18 Aug 2009 09:35:32 -0300 Subject: [PATCH] added documentation for ImagesPipeline --- docs/experimental/images.rst | 333 +++++++++++++++++++++++++++++++++++ docs/ref/settings.rst | 45 +++++ 2 files changed, 378 insertions(+) create mode 100644 docs/experimental/images.rst diff --git a/docs/experimental/images.rst b/docs/experimental/images.rst new file mode 100644 index 000000000..91dd187a9 --- /dev/null +++ b/docs/experimental/images.rst @@ -0,0 +1,333 @@ +.. _topics-images: + +.. module:: scrapy.contrib.pipeline.images + +=============== +Handling Images +=============== + +In Scrapy, the recommended way of handling image downloads is using the +:class:`ImagesPipeline`. + +This pipeline provides convenient mechanisms to download and store images and +also the following features: + +* Image format normalization (JPG) +* Image expiration +* Thumbnail creation +* Image size checking + + +Using a ImagesPipeline +======================= + +The typical workflow of working with a :class:`ImagesPipeline` goes like this: + +1. In a Spider, you obtain the URLs of the images to be downloaded and store + them in an Item. + +2. An :class:`ImagesPipeline` process the Item, downloads the images and stores + back their resulting paths in the processed Item + +We assume that if you're here you know how to handle the first part of the +workflow (if not, please refer to the tutorial), so let's focus on the second +part, using a :class:`ImagesPipeline`. + +:class:`ImagesPipeline` is a descendant of BaseImagesPipeline which in turn is +a descendant of :class:`~scrapy.contrib.pipeline.MediaPipeline`, all of this classes provide +overrideable methods, hooks and settings to customize their behaviour. + +So, for using the :class:`ImagesPipeline` you subclass it, override some +methods with custom code and set some required settings. + +The first thing we need to do is tell the pipeline where to store the +downloaded images, so set :setting:`IMAGES_DIR` to a valid directory name that +will be used for this purpose:: + + IMAGES_DIR = '/path/to/valid/dir' + +Then, as seen on the workflow, the pipeline will get the URLs of the images to +download from the item. In order to do this, you must override the +:meth:`~scrapy.contrib.pipeline.MediaPipeline.get_media_requests` method and +return a Request for each image URL:: + + def get_media_requests(self, item, info): + for image_url in item['image_urls']: + yield Request(image_url) + +Those requests will be processed by the pipeline, downloaded an when completed +the processed results will be sent to the +:meth:`~scrapy.contrib.pipeline.MediaPipeline.item_completed` method. + +The results will be a list of tuples, in wich each tuple indicates the sucess +of the downloading process and the stored image path concatenated with the +checksum of the image :: + + results = [(True, 'path#checksum'), ..., (False, Failure)] + +The :meth:`~scrapy.contrib.pipeline.MediaPipeline.item_completed` is also in +charge of returning the output value to be used as the output of the pipeline +stage, so we must return (or drop) the item as in any pipeline. + +We will override it to store the resulting image paths (passed in results) back +in the item:: + + def item_completed(self, results, item, info): + item['image_paths'] = [result.split('#')[0] for succes, result in results if succes] + + return item + +.. note:: This is a simplification of the actual process, it will be described + with more detail in upcoming sections. + +So, the complete example of our pipeline looks like this:: + + from scrapy.contrib.pipeline.images import ImagesPipeline + + class MyImagesPipeline(ImagesPipeline): + + def get_media_requests(self, item, info): + for image_url in item['image_urls']: + yield Request(image_url) + + def item_completed(self, results, item, info): + item['image_paths'] = [result.split('#')[0] for succes, result in results if succes] + + return item + +This is the most basic use of :class:`ImagesPipeline`, see upcoming sections for more details. + + +.. _topics-images-expiration: + +Image expiration +----------------- + +XXX + +.. _topics-images-thumbnails: + +Creating thumbnails +------------------- + +As mentioned in the features, :class:`ImagesPipeline` can create thumbnails of +the processed images. + +In order use this feature you must set the :attr:`~BaseImagesPipeline.THUMBS` to +a tuple of tuples, in wich each sub-tuple is a pair of thumb_id string and a +compatible python image library size (another tuple). + +See ``thumbnail`` method at http://www.pythonware.com/library/pil/handbook/image.htm. + +Example:: + + THUMBS = ( + ('50', (50, 50)), + ('110', (110, 110)), + ('270', (270, 270)) + ) + + +When you use this feature, :class:`ImagesPipeline` will create thumbnails of +the specified sizes in ``IMAGES_DIR/thumbs//.jpg``, where +```` is the ``sha1`` digest of the url of the image and +```` is the thumb_id string specified in THUMBS attribute. + +Example with previous THUMB attribute:: + + IMAGES_DIR/thumbs/image_sha1_digest/50.jpg + IMAGES_DIR/thumbs/image_sha1_digest/110.jpg + IMAGES_DIR/thumbs/image_sha1_digest/270.jpg + + +.. _topics-images-size: + +Checking image size +------------------- + +You can skip the processing of an image if its size is less than a specified +one. To use this set :setting:`IMAGES_MIN_HEIGHT` and/or +:setting:`IMAGES_MIN_WIDTH` to your likings:: + + IMAGES_MIN_HEIGHT = 270 + IMAGES_MIN_WIDTH = 270 + + +.. _ref-images: + +Reference +========= + +ImagesPipeline +-------------- + +.. class:: ImagesPipeline + + :class:`BaseImagesPipeline` descendant with filesystem support as + image's store backend + + In order to enable this pipeline you must set :setting:`IMAGES_DIR` to a + valid dirname that will be used for storing images. + + +BaseImagesPipeline +------------------ + +.. class:: BaseImagesPipeline + + :class:`~scrapy.contrib.pipeline.MediaPipeline` descendant that implements + image downloading and thumbnail generation logic. + + This pipeline tries to minimize network transfers and image processing, + doing stat of the images and determining if image is new, uptodate or + expired. + + `'new'` images are those that pipeline never processed and needs to be + downloaded from supplier site the first time. + + `'uptodate'` images are the ones that the pipeline processed and are still + valid images. + + `'expired'` images are those that pipeline already processed but the last + modification was made long time ago, so a reprocessing is recommended to + refresh it in case of change. + + :setting:`IMAGES_EXPIRES` setting controls the maximun days since an image + was modified to consider it `uptodate`. + + Downloaded images are skipped if sizes aren't greater than + :setting:`IMAGES_MIN_WIDTH` and :setting:`IMAGES_MIN_HEIGHT` limit. A proper + log messages will be printed. + + .. attribute:: THUMBS + + Thumbnail generation configuration, see :ref:`topics-images-thumbnails` + + .. method:: store_image(key, image, buf, info) + + Override this method with specific code to persist an image. + + This method is used to persist the full image and any defined + thumbnail, one a time. + + Return value is ignored. + + + .. method:: stat_key(key, info) + + Override this method with specific code to stat an image. + + This method should return and dictionary with two parameters: + + * ``last_modified``: the last modification time in seconds since the epoch + * ``checksum``: the md5sum of the content of the stored image if found + + If an exception is raised or ``last_modified`` is ``None``, then the image + will be re-downloaded. + + If the difference in days between last_modified and now is greater than + :setting:`IMAGES_EXPIRES` settings, then the image will be re-downloaded + + The checksum value is appended to returned image path after a hash sign + (#), if ``checksum`` is ``None``, then nothing is appended including the + hash sign. + + +.. module:: scrapy.contrib.pipeline + +MediaPipeline +------------- + +.. class:: MediaPipeline + + Generic pipeline that handles the media associated with an item. + + .. method:: download(request, info) + + Defines how to request the download of media. + + Default gives high priority to media requests and use scheduler, shouldn't + be necessary to override. + + This methods is called only if result for request isn't cached, request + fingerprint is used as cache key. + + + .. method:: media_to_download(request, info) + + Ongoing request hook pre-cache. + + This method is called every time a media is requested for download, and only + once for the same request because return value is cached as media result. + + Returning a non-None value implies: + + * the return value is cached and piped into :meth:`item_media_downloaded` + or :meth:`item_media_failed` + * prevents downloading, this means calling :meth:`download` method. + * :meth:`media_downloaded` or :meth:`media_failed` isn't called. + + + .. method:: get_media_requests(item, info) + + Return a list of Request objects to download for this item. + + Should return ``None`` or an iterable. + + Defaults return ``None`` (no media to download) + + + .. method:: media_downloaded(response, request, info) + + Method called on success download of media request + + Return value is cached and used as input for + :meth:`item_media_downloaded` method. Default implementation returns + ``None``. + + WARNING: returning the response object can eat your memory. + + + .. method:: media_failed(failure, request, info) + + Method called when media request failed due to any kind of download error. + + Return value is cached and used as input for :meth:`item_media_failed` method. + + Default implementation returns same Failure object. + + + .. method:: item_media_downloaded(result, item, request, info) + + Method to handle result of requested media for item. + + ``result`` is the return value of :meth:`media_downloaded` hook, or the + non-Failure instance returned by :meth:`media_failed` hook. + + Return value of this method isn't important and is recommended to return + ``None``. + + + .. method:: item_media_failed(failure, item, request, info) + + Method to handle failed result of requested media for item. + + result is the returned Failure instance of :meth:`media_failed` hook, or Failure + instance of an exception raised by :meth:`media_downloaded` hook. + + Return value of this method isn't important and is recommended to return + ``None``. + + + .. method:: item_completed(results, item, info) + + Method called when all media requests for a single item has returned a result + or failure. + + The return value of this method is used as output of pipeline stage. + + :meth:`item_completed` can return item itself or raise + :exception:`DropItem` exception. + + Default returns item + diff --git a/docs/ref/settings.rst b/docs/ref/settings.rst index 5c61b3675..bbeddafbc 100644 --- a/docs/ref/settings.rst +++ b/docs/ref/settings.rst @@ -326,6 +326,51 @@ Default: ``''`` (empty string) The module to use for pulling settings from, if the group settings is enabled. +.. setting:: IMAGES_DIR + +IMAGES_DIR +---------- + +Default: ``None`` + +Directory where :class:`ImagesPipeline` will store its images. + +For more information see :ref:`topics-images`. + +.. setting:: IMAGES_EXPIRES + +IMAGES_EXPIRES +-------------- + +Default = 90 + +Number of days for an image to be considered `expired` (downloaded again) in +:class:`ImagesPipeline`. + +For more information see :ref:`topics-images`. + +.. setting:: IMAGES_MIN_HEIGHT + +IMAGES_MIN_HEIGHT +----------------- + +Default = 0 + +Minimum height that an image is allowed to have in :class:`ImagesPipeline`. + +For more information see :ref:`topics-images-size`. + +.. setting:: IMAGES_MIN_WIDTH + +IMAGES_MIN_WIDTH +----------------- + +Default = 0 + +Minimum width that an image is allowed to have in :class:`ImagesPipeline`. + +For more information see :ref:`topics-images-size`. + .. setting:: ITEM_PIPELINES ITEM_PIPELINES