RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

added documentation for ImagesPipeline

This commit is contained in:
Ismael Carnales 2009-08-18 09:35:32 -03:00
parent ec8f934a8e
commit 1bbe7991dc
2 changed files with 378 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

333
docs/experimental/images.rst Normal file
View File

@ -0,0 +1,333 @@
.. _topics-images:
.. module:: scrapy.contrib.pipeline.images
===============
Handling Images
===============
In Scrapy, the recommended way of handling image downloads is using the
:class:`ImagesPipeline`.
This pipeline provides convenient mechanisms to download and store images and
also the following features:
* Image format normalization (JPG)
* Image expiration
* Thumbnail creation
* Image size checking
Using a ImagesPipeline
=======================
The typical workflow of working with a :class:`ImagesPipeline` goes like this:
1. In a Spider, you obtain the URLs of the images to be downloaded and store
them in an Item.
2. An :class:`ImagesPipeline` process the Item, downloads the images and stores
back their resulting paths in the processed Item
We assume that if you're here you know how to handle the first part of the
workflow (if not, please refer to the tutorial), so let's focus on the second
part, using a :class:`ImagesPipeline`.
:class:`ImagesPipeline` is a descendant of BaseImagesPipeline which in turn is
a descendant of :class:`~scrapy.contrib.pipeline.MediaPipeline`, all of this classes provide
overrideable methods, hooks and settings to customize their behaviour.
So, for using the :class:`ImagesPipeline` you subclass it, override some
methods with custom code and set some required settings.
The first thing we need to do is tell the pipeline where to store the
downloaded images, so set :setting:`IMAGES_DIR` to a valid directory name that
will be used for this purpose::
IMAGES_DIR = '/path/to/valid/dir'
Then, as seen on the workflow, the pipeline will get the URLs of the images to
download from the item. In order to do this, you must override the
:meth:`~scrapy.contrib.pipeline.MediaPipeline.get_media_requests` method and
return a Request for each image URL::
def get_media_requests(self, item, info):
for image_url in item['image_urls']:
yield Request(image_url)
Those requests will be processed by the pipeline, downloaded an when completed
the processed results will be sent to the
:meth:`~scrapy.contrib.pipeline.MediaPipeline.item_completed` method.
The results will be a list of tuples, in wich each tuple indicates the sucess
of the downloading process and the stored image path concatenated with the
checksum of the image ::
results = [(True, 'path#checksum'), ..., (False, Failure)]
The :meth:`~scrapy.contrib.pipeline.MediaPipeline.item_completed` is also in
charge of returning the output value to be used as the output of the pipeline
stage, so we must return (or drop) the item as in any pipeline.
We will override it to store the resulting image paths (passed in results) back
in the item::
def item_completed(self, results, item, info):
item['image_paths'] = [result.split('#')[0] for succes, result in results if succes]
return item
.. note:: This is a simplification of the actual process, it will be described
with more detail in upcoming sections.
So, the complete example of our pipeline looks like this::
from scrapy.contrib.pipeline.images import ImagesPipeline
class MyImagesPipeline(ImagesPipeline):
def get_media_requests(self, item, info):
for image_url in item['image_urls']:
yield Request(image_url)
def item_completed(self, results, item, info):
item['image_paths'] = [result.split('#')[0] for succes, result in results if succes]
return item
This is the most basic use of :class:`ImagesPipeline`, see upcoming sections for more details.
.. _topics-images-expiration:
Image expiration
-----------------
XXX
.. _topics-images-thumbnails:
Creating thumbnails
-------------------
As mentioned in the features, :class:`ImagesPipeline` can create thumbnails of
the processed images.
In order use this feature you must set the :attr:`~BaseImagesPipeline.THUMBS` to
a tuple of tuples, in wich each sub-tuple is a pair of thumb_id string and a
compatible python image library size (another tuple).
See ``thumbnail`` method at http://www.pythonware.com/library/pil/handbook/image.htm.
Example::
THUMBS = (
('50', (50, 50)),
('110', (110, 110)),
('270', (270, 270))
)
When you use this feature, :class:`ImagesPipeline` will create thumbnails of
the specified sizes in ``IMAGES_DIR/thumbs/<image_id>/<thumb_id>.jpg``, where
``<image_id>`` is the ``sha1`` digest of the url of the image and
``<thumb_id>`` is the thumb_id string specified in THUMBS attribute.
Example with previous THUMB attribute::
IMAGES_DIR/thumbs/image_sha1_digest/50.jpg
IMAGES_DIR/thumbs/image_sha1_digest/110.jpg
IMAGES_DIR/thumbs/image_sha1_digest/270.jpg
.. _topics-images-size:
Checking image size
-------------------
You can skip the processing of an image if its size is less than a specified
one. To use this set :setting:`IMAGES_MIN_HEIGHT` and/or
:setting:`IMAGES_MIN_WIDTH` to your likings::
IMAGES_MIN_HEIGHT = 270
IMAGES_MIN_WIDTH = 270
.. _ref-images:
Reference
=========
ImagesPipeline
--------------
.. class:: ImagesPipeline
:class:`BaseImagesPipeline` descendant with filesystem support as
image's store backend
In order to enable this pipeline you must set :setting:`IMAGES_DIR` to a
valid dirname that will be used for storing images.
BaseImagesPipeline
------------------
.. class:: BaseImagesPipeline
:class:`~scrapy.contrib.pipeline.MediaPipeline` descendant that implements
image downloading and thumbnail generation logic.
This pipeline tries to minimize network transfers and image processing,
doing stat of the images and determining if image is new, uptodate or
expired.
`'new'` images are those that pipeline never processed and needs to be
downloaded from supplier site the first time.
`'uptodate'` images are the ones that the pipeline processed and are still
valid images.
`'expired'` images are those that pipeline already processed but the last
modification was made long time ago, so a reprocessing is recommended to
refresh it in case of change.
:setting:`IMAGES_EXPIRES` setting controls the maximun days since an image
was modified to consider it `uptodate`.
Downloaded images are skipped if sizes aren't greater than
:setting:`IMAGES_MIN_WIDTH` and :setting:`IMAGES_MIN_HEIGHT` limit. A proper
log messages will be printed.
.. attribute:: THUMBS
Thumbnail generation configuration, see :ref:`topics-images-thumbnails`
.. method:: store_image(key, image, buf, info)
Override this method with specific code to persist an image.
This method is used to persist the full image and any defined
thumbnail, one a time.
Return value is ignored.
.. method:: stat_key(key, info)
Override this method with specific code to stat an image.
This method should return and dictionary with two parameters:
* ``last_modified``: the last modification time in seconds since the epoch
* ``checksum``: the md5sum of the content of the stored image if found
If an exception is raised or ``last_modified`` is ``None``, then the image
will be re-downloaded.
If the difference in days between last_modified and now is greater than
:setting:`IMAGES_EXPIRES` settings, then the image will be re-downloaded
The checksum value is appended to returned image path after a hash sign
(#), if ``checksum`` is ``None``, then nothing is appended including the
hash sign.
.. module:: scrapy.contrib.pipeline
MediaPipeline
-------------
.. class:: MediaPipeline
Generic pipeline that handles the media associated with an item.
.. method:: download(request, info)
Defines how to request the download of media.
Default gives high priority to media requests and use scheduler, shouldn't
be necessary to override.
This methods is called only if result for request isn't cached, request
fingerprint is used as cache key.
.. method:: media_to_download(request, info)
Ongoing request hook pre-cache.
This method is called every time a media is requested for download, and only
once for the same request because return value is cached as media result.
Returning a non-None value implies:
* the return value is cached and piped into :meth:`item_media_downloaded`
or :meth:`item_media_failed`
* prevents downloading, this means calling :meth:`download` method.
* :meth:`media_downloaded` or :meth:`media_failed` isn't called.
.. method:: get_media_requests(item, info)
Return a list of Request objects to download for this item.
Should return ``None`` or an iterable.
Defaults return ``None`` (no media to download)
.. method:: media_downloaded(response, request, info)
Method called on success download of media request
Return value is cached and used as input for
:meth:`item_media_downloaded` method. Default implementation returns
``None``.
WARNING: returning the response object can eat your memory.
.. method:: media_failed(failure, request, info)
Method called when media request failed due to any kind of download error.
Return value is cached and used as input for :meth:`item_media_failed` method.
Default implementation returns same Failure object.
.. method:: item_media_downloaded(result, item, request, info)
Method to handle result of requested media for item.
``result`` is the return value of :meth:`media_downloaded` hook, or the
non-Failure instance returned by :meth:`media_failed` hook.
Return value of this method isn't important and is recommended to return
``None``.
.. method:: item_media_failed(failure, item, request, info)
Method to handle failed result of requested media for item.
result is the returned Failure instance of :meth:`media_failed` hook, or Failure
instance of an exception raised by :meth:`media_downloaded` hook.
Return value of this method isn't important and is recommended to return
``None``.
.. method:: item_completed(results, item, info)
Method called when all media requests for a single item has returned a result
or failure.
The return value of this method is used as output of pipeline stage.
:meth:`item_completed` can return item itself or raise
:exception:`DropItem` exception.
Default returns item

45
docs/ref/settings.rst
View File

@ -326,6 +326,51 @@ Default: ``''`` (empty string)
The module to use for pulling settings from, if the group settings is enabled.
.. setting:: IMAGES_DIR
IMAGES_DIR
----------
Default: ``None``
Directory where :class:`ImagesPipeline` will store its images.
For more information see :ref:`topics-images`.
.. setting:: IMAGES_EXPIRES
IMAGES_EXPIRES
--------------
Default = 90
Number of days for an image to be considered `expired` (downloaded again) in
:class:`ImagesPipeline`.
For more information see :ref:`topics-images`.
.. setting:: IMAGES_MIN_HEIGHT
IMAGES_MIN_HEIGHT
-----------------
Default = 0
Minimum height that an image is allowed to have in :class:`ImagesPipeline`.
For more information see :ref:`topics-images-size`.
.. setting:: IMAGES_MIN_WIDTH
IMAGES_MIN_WIDTH
-----------------
Default = 0
Minimum width that an image is allowed to have in :class:`ImagesPipeline`.
For more information see :ref:`topics-images-size`.
.. setting:: ITEM_PIPELINES
ITEM_PIPELINES