RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/docs/experimental/images.rst

10 KiB
Raw Blame History

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> </head>

Handling Images

System Message: ERROR/3 (<stdin>, line 3)

Unknown directive type "module".

.. module:: scrapy.contrib.pipeline.images

In Scrapy, the recommended way of handling image downloads is using the :class:`ImagesPipeline`.

System Message: ERROR/3 (<stdin>, line 9); backlink

Unknown interpreted text role "class".

This pipeline provides convenient mechanisms to download and store images and also the following features:

  • Image format normalization (JPG)
  • Image expiration
  • Thumbnail creation
  • Image size checking

Using a ImagesPipeline

The typical workflow of working with a :class:`ImagesPipeline` goes like this:

System Message: ERROR/3 (<stdin>, line 24); backlink

Unknown interpreted text role "class".

  1. In a Spider, you obtain the URLs of the images to be downloaded and store them in an Item.

  2. An :class:`ImagesPipeline` process the Item, downloads the images and stores back their resulting paths in the processed Item

    System Message: ERROR/3 (<stdin>, line 29); backlink

    Unknown interpreted text role "class".

We assume that if you're here you know how to handle the first part of the workflow (if not, please refer to the tutorial), so let's focus on the second part, using a :class:`ImagesPipeline`.

System Message: ERROR/3 (<stdin>, line 32); backlink

Unknown interpreted text role "class".

:class:`ImagesPipeline` is a descendant of BaseImagesPipeline which in turn is a descendant of :class:`~scrapy.contrib.pipeline.MediaPipeline`, all of this classes provide overrideable methods, hooks and settings to customize their behaviour.

System Message: ERROR/3 (<stdin>, line 36); backlink

Unknown interpreted text role "class".

System Message: ERROR/3 (<stdin>, line 36); backlink

Unknown interpreted text role "class".

So, for using the :class:`ImagesPipeline` you subclass it, override some methods with custom code and set some required settings.

System Message: ERROR/3 (<stdin>, line 40); backlink

Unknown interpreted text role "class".

The first thing we need to do is tell the pipeline where to store the downloaded images, so set :setting:`IMAGES_DIR` to a valid directory name that will be used for this purpose:

System Message: ERROR/3 (<stdin>, line 43); backlink

Unknown interpreted text role "setting".
IMAGES_DIR = '/path/to/valid/dir'

Then, as seen on the workflow, the pipeline will get the URLs of the images to download from the item. In order to do this, you must override the :meth:`~scrapy.contrib.pipeline.MediaPipeline.get_media_requests` method and return a Request for each image URL:

System Message: ERROR/3 (<stdin>, line 49); backlink

Unknown interpreted text role "meth".
def get_media_requests(self, item, info):
    for image_url in item['image_urls']:
        yield Request(image_url)

Those requests will be processed by the pipeline, downloaded an when completed the processed results will be sent to the :meth:`~scrapy.contrib.pipeline.MediaPipeline.item_completed` method.

System Message: ERROR/3 (<stdin>, line 58); backlink

Unknown interpreted text role "meth".

The results will be a list of tuples, in wich each tuple indicates the sucess of the downloading process and the stored image path concatenated with the checksum of the image

results = [(True, 'path#checksum'), ..., (False, Failure)]

The :meth:`~scrapy.contrib.pipeline.MediaPipeline.item_completed` is also in charge of returning the output value to be used as the output of the pipeline stage, so we must return (or drop) the item as in any pipeline.

System Message: ERROR/3 (<stdin>, line 68); backlink

Unknown interpreted text role "meth".

We will override it to store the resulting image paths (passed in results) back in the item:

def item_completed(self, results, item, info):
    item['image_paths'] = [result.split('#')[0] for succes, result in results if succes]

    return item

Note

This is a simplification of the actual process, it will be described with more detail in upcoming sections.

So, the complete example of our pipeline looks like this:

from scrapy.contrib.pipeline.images import ImagesPipeline

class MyImagesPipeline(ImagesPipeline):

    def get_media_requests(self, item, info):
        for image_url in item['image_urls']:
            yield Request(image_url)

    def item_completed(self, results, item, info):
        item['image_paths'] = [result.split('#')[0] for succes, result in results if succes]

        return item

This is the most basic use of :class:`ImagesPipeline`, see upcoming sections for more details.

System Message: ERROR/3 (<stdin>, line 98); backlink

Unknown interpreted text role "class".

Image expiration

XXX

Creating thumbnails

As mentioned in the features, :class:`ImagesPipeline` can create thumbnails of the processed images.

System Message: ERROR/3 (<stdin>, line 113); backlink

Unknown interpreted text role "class".

In order use this feature you must set the :attr:`~BaseImagesPipeline.THUMBS` to a tuple of tuples, in wich each sub-tuple is a pair of thumb_id string and a compatible python image library size (another tuple).

System Message: ERROR/3 (<stdin>, line 116); backlink

Unknown interpreted text role "attr".

See thumbnail method at http://www.pythonware.com/library/pil/handbook/image.htm.

Example:

THUMBS = (
    ('50', (50, 50)),
    ('110', (110, 110)),
    ('270', (270, 270))
)

When you use this feature, :class:`ImagesPipeline` will create thumbnails of the specified sizes in IMAGES_DIR/thumbs/<image_id>/<thumb_id>.jpg, where <image_id> is the sha1 digest of the url of the image and <thumb_id> is the thumb_id string specified in THUMBS attribute.

System Message: ERROR/3 (<stdin>, line 131); backlink

Unknown interpreted text role "class".

Example with previous THUMB attribute:

IMAGES_DIR/thumbs/image_sha1_digest/50.jpg
IMAGES_DIR/thumbs/image_sha1_digest/110.jpg
IMAGES_DIR/thumbs/image_sha1_digest/270.jpg

Checking image size

You can skip the processing of an image if its size is less than a specified one. To use this set :setting:`IMAGES_MIN_HEIGHT` and/or :setting:`IMAGES_MIN_WIDTH` to your likings:

System Message: ERROR/3 (<stdin>, line 148); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 148); backlink

Unknown interpreted text role "setting".
IMAGES_MIN_HEIGHT = 270
IMAGES_MIN_WIDTH = 270

Reference

ImagesPipeline

:class:`BaseImagesPipeline` descendant with filesystem support as image's store backend

System Message: ERROR/3 (<stdin>, line 166); backlink

Unknown interpreted text role "class".

In order to enable this pipeline you must set :setting:`IMAGES_DIR` to a valid dirname that will be used for storing images.

System Message: ERROR/3 (<stdin>, line 169); backlink

Unknown interpreted text role "setting".

BaseImagesPipeline

:class:`~scrapy.contrib.pipeline.MediaPipeline` descendant that implements image downloading and thumbnail generation logic.

System Message: ERROR/3 (<stdin>, line 178); backlink

Unknown interpreted text role "class".

This pipeline tries to minimize network transfers and image processing, doing stat of the images and determining if image is new, uptodate or expired.

'new' images are those that pipeline never processed and needs to be downloaded from supplier site the first time.

'uptodate' images are the ones that the pipeline processed and are still valid images.

'expired' images are those that pipeline already processed but the last modification was made long time ago, so a reprocessing is recommended to refresh it in case of change.

:setting:`IMAGES_EXPIRES` setting controls the maximun days since an image was modified to consider it uptodate.

System Message: ERROR/3 (<stdin>, line 195); backlink

Unknown interpreted text role "setting".

Downloaded images are skipped if sizes aren't greater than :setting:`IMAGES_MIN_WIDTH` and :setting:`IMAGES_MIN_HEIGHT` limit. A proper log messages will be printed.

System Message: ERROR/3 (<stdin>, line 198); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 198); backlink

Unknown interpreted text role "setting".

System Message: ERROR/3 (<stdin>, line 202)

Unknown directive type "attribute".

.. attribute:: THUMBS

   Thumbnail generation configuration, see :ref:`topics-images-thumbnails`

System Message: ERROR/3 (<stdin>, line 206)

Unknown directive type "method".

.. method:: store_image(key, image, buf, info)

   Override this method with specific code to persist an image.

   This method is used to persist the full image and any defined
   thumbnail, one a time.

   Return value is ignored.

System Message: ERROR/3 (<stdin>, line 216)

Unknown directive type "method".

.. method:: stat_key(key, info)

   Override this method with specific code to stat an image.

   This method should return and dictionary with two parameters:

   * ``last_modified``: the last modification time in seconds since the epoch
   * ``checksum``: the md5sum of the content of the stored image if found

   If an exception is raised or ``last_modified`` is ``None``, then the image
   will be re-downloaded.

   If the difference in days between last_modified and now is greater than
   :setting:`IMAGES_EXPIRES` settings, then the image will be re-downloaded

   The checksum value is appended to returned image path after a hash sign
   (#), if ``checksum`` is ``None``, then nothing is appended including the
   hash sign.

System Message: ERROR/3 (<stdin>, line 236)

Unknown directive type "module".

.. module:: scrapy.contrib.pipeline

MediaPipeline

Generic pipeline that handles the media associated with an item.

System Message: ERROR/3 (<stdin>, line 245)

Unknown directive type "method".

.. method:: download(request, info)

   Defines how to request the download of media.

   Default gives high priority to media requests and use scheduler, shouldn't
   be necessary to override.

   This methods is called only if result for request isn't cached, request
   fingerprint is used as cache key.

System Message: ERROR/3 (<stdin>, line 256)

Unknown directive type "method".

.. method:: media_to_download(request, info)

   Ongoing request hook pre-cache.

   This method is called every time a media is requested for download, and only
   once for the same request because return value is cached as media result.

   Returning a non-None value implies:

   * the return value is cached and piped into :meth:`item_media_downloaded`
     or :meth:`item_media_failed`
   * prevents downloading, this means calling :meth:`download` method.
   * :meth:`media_downloaded` or :meth:`media_failed` isn't called.

System Message: ERROR/3 (<stdin>, line 271)

Unknown directive type "method".

.. method:: get_media_requests(item, info)

   Return a list of Request objects to download for this item.

   Should return ``None`` or an iterable.

   Defaults return ``None`` (no media to download)

System Message: ERROR/3 (<stdin>, line 280)

Unknown directive type "method".

.. method:: media_downloaded(response, request, info)

   Method called on success download of media request

   Return value is cached and used as input for
   :meth:`item_media_downloaded` method.  Default implementation returns
   ``None``.

   WARNING: returning the response object can eat your memory.

System Message: ERROR/3 (<stdin>, line 291)

Unknown directive type "method".

.. method:: media_failed(failure, request, info)

   Method called when media request failed due to any kind of download error.

   Return value is cached and used as input for :meth:`item_media_failed` method.

   Default implementation returns same Failure object.

System Message: ERROR/3 (<stdin>, line 300)

Unknown directive type "method".

.. method:: item_media_downloaded(result, item, request, info)

   Method to handle result of requested media for item.

   ``result`` is the return value of :meth:`media_downloaded` hook, or the
   non-Failure instance returned by :meth:`media_failed` hook.

   Return value of this method isn't important and is recommended to return
   ``None``.

System Message: ERROR/3 (<stdin>, line 311)

Unknown directive type "method".

.. method:: item_media_failed(failure, item, request, info)

   Method to handle failed result of requested media for item.

   result is the returned Failure instance of :meth:`media_failed` hook, or Failure
   instance of an exception raised by :meth:`media_downloaded` hook.

   Return value of this method isn't important and is recommended to return
   ``None``.

System Message: ERROR/3 (<stdin>, line 322)

Unknown directive type "method".

.. method:: item_completed(results, item, info)

   Method called when all media requests for a single item has returned a result
   or failure.

   The return value of this method is used as output of pipeline stage.

   :meth:`item_completed` can return item itself or raise
   :exception:`DropItem` exception.

   Default returns item
</html>