mirror of https://github.com/scrapy/scrapy.git
Merge pull request #1150 from eliasdorneles/docs-files-pipeline
Documenting Files Pipeline together with Images Pipeline
This commit is contained in:
Pablo Hoffman
commit
e4122cdd79
|
|
@ -148,7 +148,7 @@ Solving specific problems
|
|||
topics/firefox
|
||||
topics/firebug
|
||||
topics/leaks
|
||||
topics/images
|
||||
topics/media-pipeline
|
||||
topics/ubuntu
|
||||
topics/deploy
|
||||
topics/autothrottle
|
||||
|
|
@ -180,8 +180,8 @@ Solving specific problems
|
|||
:doc:`topics/leaks`
|
||||
Learn how to find and get rid of memory leaks in your crawler.
|
||||
|
||||
:doc:`topics/images`
|
||||
Download static images associated with your scraped items.
|
||||
:doc:`topics/media-pipeline`
|
||||
Download files and/or images associated with your scraped items.
|
||||
|
||||
:doc:`topics/ubuntu`
|
||||
Install latest Scrapy packages easily on Ubuntu
|
||||
|
|
|
|||
|
|
@ -1,328 +0,0 @@
|
|||
.. _topics-images:
|
||||
|
||||
=======================
|
||||
Downloading Item Images
|
||||
=======================
|
||||
|
||||
.. currentmodule:: scrapy.contrib.pipeline.images
|
||||
|
||||
Scrapy provides an :doc:`item pipeline </topics/item-pipeline>` for downloading
|
||||
images attached to a particular item, for example, when you scrape products and
|
||||
also want to download their images locally.
|
||||
|
||||
This pipeline, called the Images Pipeline and implemented in the
|
||||
:class:`ImagesPipeline` class, provides a convenient way for
|
||||
downloading and storing images locally with some additional features:
|
||||
|
||||
* Convert all downloaded images to a common format (JPG) and mode (RGB)
|
||||
* Avoid re-downloading images which were downloaded recently
|
||||
* Thumbnail generation
|
||||
* Check images width/height to make sure they meet a minimum constraint
|
||||
|
||||
This pipeline also keeps an internal queue of those images which are currently
|
||||
being scheduled for download, and connects those items that arrive containing
|
||||
the same image, to that queue. This avoids downloading the same image more than
|
||||
once when it's shared by several items.
|
||||
|
||||
`Pillow`_ is used for thumbnailing and normalizing images to JPEG/RGB format,
|
||||
so you need to install this library in order to use the images pipeline.
|
||||
`Python Imaging Library`_ (PIL) should also work in most cases, but it
|
||||
is known to cause troubles in some setups, so we recommend to use `Pillow`_
|
||||
instead of `PIL <Python Imaging Library>`_.
|
||||
|
||||
.. _Pillow: https://github.com/python-pillow/Pillow
|
||||
.. _Python Imaging Library: http://www.pythonware.com/products/pil/
|
||||
|
||||
Using the Images Pipeline
|
||||
=========================
|
||||
|
||||
The typical workflow, when using the :class:`ImagesPipeline` goes like
|
||||
this:
|
||||
|
||||
1. In a Spider, you scrape an item and put the URLs of its images into a
|
||||
``image_urls`` field.
|
||||
|
||||
2. The item is returned from the spider and goes to the item pipeline.
|
||||
|
||||
3. When the item reaches the :class:`ImagesPipeline`, the URLs in the
|
||||
``image_urls`` field are scheduled for download using the standard
|
||||
Scrapy scheduler and downloader (which means the scheduler and downloader
|
||||
middlewares are reused), but with a higher priority, processing them before other
|
||||
pages are scraped. The item remains "locked" at that particular pipeline stage
|
||||
until the images have finish downloading (or fail for some reason).
|
||||
|
||||
4. When the images are downloaded another field (``images``) will be populated
|
||||
with the results. This field will contain a list of dicts with information
|
||||
about the images downloaded, such as the downloaded path, the original
|
||||
scraped url (taken from the ``image_urls`` field) , and the image checksum.
|
||||
The images in the list of the ``images`` field will retain the same order of
|
||||
the original ``image_urls`` field. If some image failed downloading, an
|
||||
error will be logged and the image won't be present in the ``images`` field.
|
||||
|
||||
|
||||
Usage example
|
||||
=============
|
||||
|
||||
In order to use the image pipeline first
|
||||
:ref:`enable it <topics-images-enabling>`.
|
||||
|
||||
Then, if a spider returns a dict with 'image_urls' key,
|
||||
the pipeline will put the results under 'images' key.
|
||||
|
||||
If you prefer to use :class:`~.Item` then define a custom
|
||||
item with the ``image_urls`` and ``images`` fields::
|
||||
|
||||
import scrapy
|
||||
|
||||
class MyItem(scrapy.Item):
|
||||
|
||||
# ... other item fields ...
|
||||
image_urls = scrapy.Field()
|
||||
images = scrapy.Field()
|
||||
|
||||
If you need something more complex and want to override the custom images
|
||||
pipeline behaviour, see :ref:`topics-images-override`.
|
||||
|
||||
.. _topics-images-enabling:
|
||||
|
||||
Enabling your Images Pipeline
|
||||
=============================
|
||||
|
||||
.. setting:: IMAGES_STORE
|
||||
|
||||
To enable your images pipeline you must first add it to your project
|
||||
:setting:`ITEM_PIPELINES` setting::
|
||||
|
||||
ITEM_PIPELINES = {'scrapy.contrib.pipeline.images.ImagesPipeline': 1}
|
||||
|
||||
And set the :setting:`IMAGES_STORE` setting to a valid directory that will be
|
||||
used for storing the downloaded images. Otherwise the pipeline will remain
|
||||
disabled, even if you include it in the :setting:`ITEM_PIPELINES` setting.
|
||||
|
||||
For example::
|
||||
|
||||
IMAGES_STORE = '/path/to/valid/dir'
|
||||
|
||||
Images Storage
|
||||
==============
|
||||
|
||||
File system is currently the only officially supported storage, but there is
|
||||
also (undocumented) support for `Amazon S3`_.
|
||||
|
||||
.. _Amazon S3: http://aws.amazon.com/s3/
|
||||
|
||||
File system storage
|
||||
-------------------
|
||||
|
||||
The images are stored in files (one per image), using a `SHA1 hash`_ of their
|
||||
URLs for the file names.
|
||||
|
||||
For example, the following image URL::
|
||||
|
||||
http://www.example.com/image.jpg
|
||||
|
||||
Whose `SHA1 hash` is::
|
||||
|
||||
3afec3b4765f8f0a07b78f98c07b83f013567a0a
|
||||
|
||||
Will be downloaded and stored in the following file::
|
||||
|
||||
<IMAGES_STORE>/full/3afec3b4765f8f0a07b78f98c07b83f013567a0a.jpg
|
||||
|
||||
Where:
|
||||
|
||||
* ``<IMAGES_STORE>`` is the directory defined in :setting:`IMAGES_STORE` setting
|
||||
|
||||
* ``full`` is a sub-directory to separate full images from thumbnails (if
|
||||
used). For more info see :ref:`topics-images-thumbnails`.
|
||||
|
||||
Additional features
|
||||
===================
|
||||
|
||||
Image expiration
|
||||
----------------
|
||||
|
||||
.. setting:: IMAGES_EXPIRES
|
||||
|
||||
The Image Pipeline avoids downloading images that were downloaded recently. To
|
||||
adjust this retention delay use the :setting:`IMAGES_EXPIRES` setting, which
|
||||
specifies the delay in number of days::
|
||||
|
||||
# 90 days of delay for image expiration
|
||||
IMAGES_EXPIRES = 90
|
||||
|
||||
.. _topics-images-thumbnails:
|
||||
|
||||
Thumbnail generation
|
||||
--------------------
|
||||
|
||||
The Images Pipeline can automatically create thumbnails of the downloaded
|
||||
images.
|
||||
|
||||
.. setting:: IMAGES_THUMBS
|
||||
|
||||
In order use this feature, you must set :setting:`IMAGES_THUMBS` to a dictionary
|
||||
where the keys are the thumbnail names and the values are their dimensions.
|
||||
|
||||
For example::
|
||||
|
||||
IMAGES_THUMBS = {
|
||||
'small': (50, 50),
|
||||
'big': (270, 270),
|
||||
}
|
||||
|
||||
When you use this feature, the Images Pipeline will create thumbnails of the
|
||||
each specified size with this format::
|
||||
|
||||
<IMAGES_STORE>/thumbs/<size_name>/<image_id>.jpg
|
||||
|
||||
Where:
|
||||
|
||||
* ``<size_name>`` is the one specified in the :setting:`IMAGES_THUMBS`
|
||||
dictionary keys (``small``, ``big``, etc)
|
||||
|
||||
* ``<image_id>`` is the `SHA1 hash`_ of the image url
|
||||
|
||||
.. _SHA1 hash: http://en.wikipedia.org/wiki/SHA_hash_functions
|
||||
|
||||
Example of image files stored using ``small`` and ``big`` thumbnail names::
|
||||
|
||||
<IMAGES_STORE>/full/63bbfea82b8880ed33cdb762aa11fab722a90a24.jpg
|
||||
<IMAGES_STORE>/thumbs/small/63bbfea82b8880ed33cdb762aa11fab722a90a24.jpg
|
||||
<IMAGES_STORE>/thumbs/big/63bbfea82b8880ed33cdb762aa11fab722a90a24.jpg
|
||||
|
||||
The first one is the full image, as downloaded from the site.
|
||||
|
||||
Filtering out small images
|
||||
--------------------------
|
||||
|
||||
.. setting:: IMAGES_MIN_HEIGHT
|
||||
|
||||
.. setting:: IMAGES_MIN_WIDTH
|
||||
|
||||
You can drop images which are too small, by specifying the minimum allowed size
|
||||
in the :setting:`IMAGES_MIN_HEIGHT` and :setting:`IMAGES_MIN_WIDTH` settings.
|
||||
|
||||
For example::
|
||||
|
||||
IMAGES_MIN_HEIGHT = 110
|
||||
IMAGES_MIN_WIDTH = 110
|
||||
|
||||
Note: these size constraints don't affect thumbnail generation at all.
|
||||
|
||||
By default, there are no size constraints, so all images are processed.
|
||||
|
||||
.. _topics-images-override:
|
||||
|
||||
Implementing your custom Images Pipeline
|
||||
========================================
|
||||
|
||||
.. module:: scrapy.contrib.pipeline.images
|
||||
:synopsis: Images Pipeline
|
||||
|
||||
Here are the methods that you should override in your custom Images Pipeline:
|
||||
|
||||
.. class:: ImagesPipeline
|
||||
|
||||
.. method:: get_media_requests(item, info)
|
||||
|
||||
As seen on the workflow, the pipeline will get the URLs of the images to
|
||||
download from the item. In order to do this, you must override the
|
||||
:meth:`~get_media_requests` method and return a Request for each
|
||||
image URL::
|
||||
|
||||
def get_media_requests(self, item, info):
|
||||
for image_url in item['image_urls']:
|
||||
yield scrapy.Request(image_url)
|
||||
|
||||
Those requests will be processed by the pipeline and, when they have finished
|
||||
downloading, the results will be sent to the
|
||||
:meth:`~item_completed` method, as a list of 2-element tuples.
|
||||
Each tuple will contain ``(success, image_info_or_failure)`` where:
|
||||
|
||||
* ``success`` is a boolean which is ``True`` if the image was downloaded
|
||||
successfully or ``False`` if it failed for some reason
|
||||
|
||||
* ``image_info_or_error`` is a dict containing the following keys (if success
|
||||
is ``True``) or a `Twisted Failure`_ if there was a problem.
|
||||
|
||||
* ``url`` - the url where the image was downloaded from. This is the url of
|
||||
the request returned from the :meth:`~get_media_requests`
|
||||
method.
|
||||
|
||||
* ``path`` - the path (relative to :setting:`IMAGES_STORE`) where the image
|
||||
was stored
|
||||
|
||||
* ``checksum`` - a `MD5 hash`_ of the image contents
|
||||
|
||||
The list of tuples received by :meth:`~item_completed` is
|
||||
guaranteed to retain the same order of the requests returned from the
|
||||
:meth:`~get_media_requests` method.
|
||||
|
||||
Here's a typical value of the ``results`` argument::
|
||||
|
||||
[(True,
|
||||
{'checksum': '2b00042f7481c7b056c4b410d28f33cf',
|
||||
'path': 'full/7d97e98f8af710c7e7fe703abc8f639e0ee507c4.jpg',
|
||||
'url': 'http://www.example.com/images/product1.jpg'}),
|
||||
(True,
|
||||
{'checksum': 'b9628c4ab9b595f72f280b90c4fd093d',
|
||||
'path': 'full/1ca5879492b8fd606df1964ea3c1e2f4520f076f.jpg',
|
||||
'url': 'http://www.example.com/images/product2.jpg'}),
|
||||
(False,
|
||||
Failure(...))]
|
||||
|
||||
By default the :meth:`get_media_requests` method returns ``None`` which
|
||||
means there are no images to download for the item.
|
||||
|
||||
.. method:: item_completed(results, items, info)
|
||||
|
||||
The :meth:`ImagesPipeline.item_completed` method called when all image
|
||||
requests for a single item have completed (either finished downloading, or
|
||||
failed for some reason).
|
||||
|
||||
The :meth:`~item_completed` method must return the
|
||||
output that will be sent to subsequent item pipeline stages, so you must
|
||||
return (or drop) the item, as you would in any pipeline.
|
||||
|
||||
Here is an example of the :meth:`~item_completed` method where we
|
||||
store the downloaded image paths (passed in results) in the ``image_paths``
|
||||
item field, and we drop the item if it doesn't contain any images::
|
||||
|
||||
from scrapy.exceptions import DropItem
|
||||
|
||||
def item_completed(self, results, item, info):
|
||||
image_paths = [x['path'] for ok, x in results if ok]
|
||||
if not image_paths:
|
||||
raise DropItem("Item contains no images")
|
||||
item['image_paths'] = image_paths
|
||||
return item
|
||||
|
||||
By default, the :meth:`item_completed` method returns the item.
|
||||
|
||||
|
||||
Custom Images pipeline example
|
||||
==============================
|
||||
|
||||
Here is a full example of the Images Pipeline whose methods are examplified
|
||||
above::
|
||||
|
||||
import scrapy
|
||||
from scrapy.contrib.pipeline.images import ImagesPipeline
|
||||
from scrapy.exceptions import DropItem
|
||||
|
||||
class MyImagesPipeline(ImagesPipeline):
|
||||
|
||||
def get_media_requests(self, item, info):
|
||||
for image_url in item['image_urls']:
|
||||
yield scrapy.Request(image_url)
|
||||
|
||||
def item_completed(self, results, item, info):
|
||||
image_paths = [x['path'] for ok, x in results if ok]
|
||||
if not image_paths:
|
||||
raise DropItem("Item contains no images")
|
||||
item['image_paths'] = image_paths
|
||||
return item
|
||||
|
||||
.. _Twisted Failure: http://twistedmatrix.com/documents/current/api/twisted.python.failure.Failure.html
|
||||
.. _MD5 hash: http://en.wikipedia.org/wiki/MD5
|
||||
|
|
@ -0,0 +1,394 @@
|
|||
.. _topics-media-pipeline:
|
||||
|
||||
===========================================
|
||||
Downloading and processing files and images
|
||||
===========================================
|
||||
|
||||
.. currentmodule:: scrapy.contrib.pipeline.images
|
||||
|
||||
Scrapy provides reusable :doc:`item pipelines </topics/item-pipeline>` for
|
||||
downloading fies attached to a particular item (for example, when you scrape
|
||||
products and also want to download their images locally). These pipelines share
|
||||
a bit of functionality and structure (we refer to them as media pipelines), but
|
||||
typically you'll either use the Files Pipeline or the Images Pipeline.
|
||||
|
||||
Both pipelines implement these features:
|
||||
|
||||
* Avoid re-downloading media that was downloaded recently
|
||||
* Specifying where to store the media (filesystem directory, Amazon S3 bucket)
|
||||
|
||||
The Images Pipeline has a few extra functions for processing images:
|
||||
|
||||
* Convert all downloaded images to a common format (JPG) and mode (RGB)
|
||||
* Thumbnail generation
|
||||
* Check images width/height to make sure they meet a minimum constraint
|
||||
|
||||
The pipelines also keep an internal queue of those media URLs which are currently
|
||||
being scheduled for download, and connect those responses that arrive containing
|
||||
the same media to that queue. This avoids downloading the same media more than
|
||||
once when it's shared by several items.
|
||||
|
||||
Using the Files Pipeline
|
||||
=========================
|
||||
|
||||
The typical workflow, when using the :class:`FilesPipeline` goes like
|
||||
this:
|
||||
|
||||
1. In a Spider, you scrape an item and put the URLs of the desired into a
|
||||
``file_urls`` field.
|
||||
|
||||
2. The item is returned from the spider and goes to the item pipeline.
|
||||
|
||||
3. When the item reaches the :class:`FilesPipeline`, the URLs in the
|
||||
``file_urls`` field are scheduled for download using the standard
|
||||
Scrapy scheduler and downloader (which means the scheduler and downloader
|
||||
middlewares are reused), but with a higher priority, processing them before other
|
||||
pages are scraped. The item remains "locked" at that particular pipeline stage
|
||||
until the files have finish downloading (or fail for some reason).
|
||||
|
||||
4. When the files are downloaded, another field (``files``) will be populated
|
||||
with the results. This field will contain a list of dicts with information
|
||||
about the downloaded files, such as the downloaded path, the original
|
||||
scraped url (taken from the ``file_urls`` field) , and the file checksum.
|
||||
The files in the list of the ``files`` field will retain the same order of
|
||||
the original ``file_urls`` field. If some file failed downloading, an
|
||||
error will be logged and the file won't be present in the ``files`` field.
|
||||
|
||||
|
||||
Using the Images Pipeline
|
||||
=========================
|
||||
|
||||
Using the :class:`ImagesPipeline` is a lot like using the :class:`FilesPipeline`,
|
||||
except the default field names used are different: you use ``image_urls`` for
|
||||
the image URLs of an item and it will populate an ``images`` field for the information
|
||||
about the downloaded images.
|
||||
|
||||
The advantage of using the :class:`ImagesPipeline` for image files is that you
|
||||
can configure some extra functions like generating thumbnails and filtering
|
||||
the images based on their size.
|
||||
|
||||
The Images Pipeline uses `Pillow`_ for thumbnailing and normalizing images to
|
||||
JPEG/RGB format, so you need to install this library in order to use it.
|
||||
`Python Imaging Library`_ (PIL) should also work in most cases, but it is known
|
||||
to cause troubles in some setups, so we recommend to use `Pillow`_ instead of
|
||||
`PIL <Python Imaging Library>`_.
|
||||
|
||||
.. _Pillow: https://github.com/python-pillow/Pillow
|
||||
.. _Python Imaging Library: http://www.pythonware.com/products/pil/
|
||||
|
||||
|
||||
Usage example
|
||||
=============
|
||||
|
||||
In order to use a media pipeline first, :ref:`enable it
|
||||
<topics-media-pipeline-enabling>`.
|
||||
|
||||
Then, if a spider returns a dict with the URLs key ('file_urls' or
|
||||
'image_urls', for the Files or Images Pipeline respectively), the pipeline will
|
||||
put the results under respective key ('files' or images').
|
||||
|
||||
If you prefer to use :class:`~.Item`, then define a custom item with the
|
||||
necessary fields, like in this example for Images Pipeline::
|
||||
|
||||
import scrapy
|
||||
|
||||
class MyItem(scrapy.Item):
|
||||
|
||||
# ... other item fields ...
|
||||
image_urls = scrapy.Field()
|
||||
images = scrapy.Field()
|
||||
|
||||
If you need something more complex and want to override the custom pipeline
|
||||
behaviour, see :ref:`topics-media-pipeline-override`.
|
||||
|
||||
.. _topics-media-pipeline-enabling:
|
||||
|
||||
Enabling your Media Pipeline
|
||||
=============================
|
||||
|
||||
.. setting:: IMAGES_STORE
|
||||
.. setting:: FILES_STORE
|
||||
|
||||
To enable your media pipeline you must first add it to your project
|
||||
:setting:`ITEM_PIPELINES` setting.
|
||||
|
||||
For Images Pipeline, use::
|
||||
|
||||
ITEM_PIPELINES = {'scrapy.contrib.pipeline.images.ImagesPipeline': 1}
|
||||
|
||||
For Files Pipeline, use::
|
||||
|
||||
ITEM_PIPELINES = {'scrapy.contrib.pipeline.files.FilesPipeline': 1}
|
||||
|
||||
|
||||
.. note::
|
||||
You can also use both the Files and Images Pipeline at the same time.
|
||||
|
||||
|
||||
Then, configure the target storage setting to a valid value that will be used
|
||||
for storing the downloaded images. Otherwise the pipeline will remain disabled,
|
||||
even if you include it in the :setting:`ITEM_PIPELINES` setting.
|
||||
|
||||
For the Files Pipeline, set the :setting:`FILES_STORE` setting::
|
||||
|
||||
FILES_STORE = '/path/to/valid/dir'
|
||||
|
||||
For the Images Pipeline, set the :setting:`IMAGES_STORE` setting::
|
||||
|
||||
IMAGES_STORE = '/path/to/valid/dir'
|
||||
|
||||
Supported Storage
|
||||
=================
|
||||
|
||||
File system is currently the only officially supported storage, but there is
|
||||
also (undocumented) support for storing files in `Amazon S3`_.
|
||||
|
||||
.. _Amazon S3: http://aws.amazon.com/s3/
|
||||
|
||||
File system storage
|
||||
-------------------
|
||||
|
||||
The files are stored using a `SHA1 hash`_ of their URLs for the file names.
|
||||
|
||||
For example, the following image URL::
|
||||
|
||||
http://www.example.com/image.jpg
|
||||
|
||||
Whose `SHA1 hash` is::
|
||||
|
||||
3afec3b4765f8f0a07b78f98c07b83f013567a0a
|
||||
|
||||
Will be downloaded and stored in the following file::
|
||||
|
||||
<IMAGES_STORE>/full/3afec3b4765f8f0a07b78f98c07b83f013567a0a.jpg
|
||||
|
||||
Where:
|
||||
|
||||
* ``<IMAGES_STORE>`` is the directory defined in :setting:`IMAGES_STORE` setting
|
||||
for the Images Pipeline.
|
||||
|
||||
* ``full`` is a sub-directory to separate full images from thumbnails (if
|
||||
used). For more info see :ref:`topics-images-thumbnails`.
|
||||
|
||||
|
||||
Additional features
|
||||
===================
|
||||
|
||||
File expiration
|
||||
----------------
|
||||
|
||||
.. setting:: IMAGES_EXPIRES
|
||||
.. setting:: FILES_EXPIRES
|
||||
|
||||
The Image Pipeline avoids downloading files that were downloaded recently. To
|
||||
adjust this retention delay use the :setting:`FILES_EXPIRES` setting (or
|
||||
:setting:`IMAGES_EXPIRES`, in case of Images Pipeline), which
|
||||
specifies the delay in number of days::
|
||||
|
||||
# 90 days of delay for files expiration
|
||||
FILES_EXPIRES = 90
|
||||
|
||||
# 30 days of delay for images expiration
|
||||
IMAGES_EXPIRES = 30
|
||||
|
||||
.. _topics-images-thumbnails:
|
||||
|
||||
Thumbnail generation for images
|
||||
-------------------------------
|
||||
|
||||
The Images Pipeline can automatically create thumbnails of the downloaded
|
||||
images.
|
||||
|
||||
.. setting:: IMAGES_THUMBS
|
||||
|
||||
In order use this feature, you must set :setting:`IMAGES_THUMBS` to a dictionary
|
||||
where the keys are the thumbnail names and the values are their dimensions.
|
||||
|
||||
For example::
|
||||
|
||||
IMAGES_THUMBS = {
|
||||
'small': (50, 50),
|
||||
'big': (270, 270),
|
||||
}
|
||||
|
||||
When you use this feature, the Images Pipeline will create thumbnails of the
|
||||
each specified size with this format::
|
||||
|
||||
<IMAGES_STORE>/thumbs/<size_name>/<image_id>.jpg
|
||||
|
||||
Where:
|
||||
|
||||
* ``<size_name>`` is the one specified in the :setting:`IMAGES_THUMBS`
|
||||
dictionary keys (``small``, ``big``, etc)
|
||||
|
||||
* ``<image_id>`` is the `SHA1 hash`_ of the image url
|
||||
|
||||
.. _SHA1 hash: http://en.wikipedia.org/wiki/SHA_hash_functions
|
||||
|
||||
Example of image files stored using ``small`` and ``big`` thumbnail names::
|
||||
|
||||
<IMAGES_STORE>/full/63bbfea82b8880ed33cdb762aa11fab722a90a24.jpg
|
||||
<IMAGES_STORE>/thumbs/small/63bbfea82b8880ed33cdb762aa11fab722a90a24.jpg
|
||||
<IMAGES_STORE>/thumbs/big/63bbfea82b8880ed33cdb762aa11fab722a90a24.jpg
|
||||
|
||||
The first one is the full image, as downloaded from the site.
|
||||
|
||||
Filtering out small images
|
||||
--------------------------
|
||||
|
||||
.. setting:: IMAGES_MIN_HEIGHT
|
||||
|
||||
.. setting:: IMAGES_MIN_WIDTH
|
||||
|
||||
When using the Images Pipeline, you can drop images which are too small, by
|
||||
specifying the minimum allowed size in the :setting:`IMAGES_MIN_HEIGHT` and
|
||||
:setting:`IMAGES_MIN_WIDTH` settings.
|
||||
|
||||
For example::
|
||||
|
||||
IMAGES_MIN_HEIGHT = 110
|
||||
IMAGES_MIN_WIDTH = 110
|
||||
|
||||
Note: these size constraints don't affect thumbnail generation at all.
|
||||
|
||||
By default, there are no size constraints, so all images are processed.
|
||||
|
||||
.. _topics-media-pipeline-override:
|
||||
|
||||
Extending the Media Pipelines
|
||||
=============================
|
||||
|
||||
.. module:: scrapy.contrib.pipeline.files
|
||||
:synopsis: Files Pipeline
|
||||
|
||||
See here the methods that you can override in your custom Files Pipeline:
|
||||
|
||||
.. class:: FilesPipeline
|
||||
|
||||
.. method:: FilesPipeline.get_media_requests(item, info)
|
||||
|
||||
As seen on the workflow, the pipeline will get the URLs of the images to
|
||||
download from the item. In order to do this, you can override the
|
||||
:meth:`~get_media_requests` method and return a Request for each
|
||||
file URL::
|
||||
|
||||
def get_media_requests(self, item, info):
|
||||
for file_url in item['file_urls']:
|
||||
yield scrapy.Request(file_url)
|
||||
|
||||
Those requests will be processed by the pipeline and, when they have finished
|
||||
downloading, the results will be sent to the
|
||||
:meth:`~item_completed` method, as a list of 2-element tuples.
|
||||
Each tuple will contain ``(success, file_info_or_error)`` where:
|
||||
|
||||
* ``success`` is a boolean which is ``True`` if the image was downloaded
|
||||
successfully or ``False`` if it failed for some reason
|
||||
|
||||
* ``file_info_or_error`` is a dict containing the following keys (if success
|
||||
is ``True``) or a `Twisted Failure`_ if there was a problem.
|
||||
|
||||
* ``url`` - the url where the file was downloaded from. This is the url of
|
||||
the request returned from the :meth:`~get_media_requests`
|
||||
method.
|
||||
|
||||
* ``path`` - the path (relative to :setting:`FILES_STORE`) where the file
|
||||
was stored
|
||||
|
||||
* ``checksum`` - a `MD5 hash`_ of the image contents
|
||||
|
||||
The list of tuples received by :meth:`~item_completed` is
|
||||
guaranteed to retain the same order of the requests returned from the
|
||||
:meth:`~get_media_requests` method.
|
||||
|
||||
Here's a typical value of the ``results`` argument::
|
||||
|
||||
[(True,
|
||||
{'checksum': '2b00042f7481c7b056c4b410d28f33cf',
|
||||
'path': 'full/0a79c461a4062ac383dc4fade7bc09f1384a3910.jpg',
|
||||
'url': 'http://www.example.com/files/product1.pdf'}),
|
||||
(False,
|
||||
Failure(...))]
|
||||
|
||||
By default the :meth:`get_media_requests` method returns ``None`` which
|
||||
means there are no files to download for the item.
|
||||
|
||||
.. method:: FilesPipeline.item_completed(results, items, info)
|
||||
|
||||
The :meth:`FilesPipeline.item_completed` method called when all file
|
||||
requests for a single item have completed (either finished downloading, or
|
||||
failed for some reason).
|
||||
|
||||
The :meth:`~item_completed` method must return the
|
||||
output that will be sent to subsequent item pipeline stages, so you must
|
||||
return (or drop) the item, as you would in any pipeline.
|
||||
|
||||
Here is an example of the :meth:`~item_completed` method where we
|
||||
store the downloaded file paths (passed in results) in the ``file_paths``
|
||||
item field, and we drop the item if it doesn't contain any files::
|
||||
|
||||
from scrapy.exceptions import DropItem
|
||||
|
||||
def item_completed(self, results, item, info):
|
||||
file_paths = [x['path'] for ok, x in results if ok]
|
||||
if not file_paths:
|
||||
raise DropItem("Item contains no files")
|
||||
item['file_paths'] = file_paths
|
||||
return item
|
||||
|
||||
By default, the :meth:`item_completed` method returns the item.
|
||||
|
||||
|
||||
.. module:: scrapy.contrib.pipeline.images
|
||||
:synopsis: Images Pipeline
|
||||
|
||||
See here the methods that you can override in your custom Images Pipeline:
|
||||
|
||||
.. class:: ImagesPipeline
|
||||
|
||||
The :class:`ImagesPipeline` is an extension of the :class:`FilesPipeline`,
|
||||
customizing the field names and adding custom behavior for images.
|
||||
|
||||
.. method:: ImagesPipeline.get_media_requests(item, info)
|
||||
|
||||
Works the same way as :meth:`FilesPipeline.get_media_requests` method,
|
||||
but using a different field name for image urls.
|
||||
|
||||
Must return a Request for each image URL.
|
||||
|
||||
.. method:: ImagesPipeline.item_completed(results, items, info)
|
||||
|
||||
The :meth:`ImagesPipeline.item_completed` method is called when all image
|
||||
requests for a single item have completed (either finished downloading, or
|
||||
failed for some reason).
|
||||
|
||||
Works the same way as :meth:`FilesPipeline.item_completed` method,
|
||||
but using a different field names for storing image downloading results.
|
||||
|
||||
By default, the :meth:`item_completed` method returns the item.
|
||||
|
||||
|
||||
Custom Images pipeline example
|
||||
==============================
|
||||
|
||||
Here is a full example of the Images Pipeline whose methods are examplified
|
||||
above::
|
||||
|
||||
import scrapy
|
||||
from scrapy.contrib.pipeline.images import ImagesPipeline
|
||||
from scrapy.exceptions import DropItem
|
||||
|
||||
class MyImagesPipeline(ImagesPipeline):
|
||||
|
||||
def get_media_requests(self, item, info):
|
||||
for image_url in item['image_urls']:
|
||||
yield scrapy.Request(image_url)
|
||||
|
||||
def item_completed(self, results, item, info):
|
||||
image_paths = [x['path'] for ok, x in results if ok]
|
||||
if not image_paths:
|
||||
raise DropItem("Item contains no images")
|
||||
item['image_paths'] = image_paths
|
||||
return item
|
||||
|
||||
.. _Twisted Failure: http://twistedmatrix.com/documents/current/api/twisted.python.failure.Failure.html
|
||||
.. _MD5 hash: http://en.wikipedia.org/wiki/MD5
|
||||
|
|
@ -1,5 +1,7 @@
|
|||
"""
|
||||
Files Pipeline
|
||||
|
||||
See documentation in topics/media-pipeline.rst
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
"""
|
||||
Images Pipeline
|
||||
|
||||
See documentation in topics/images.rst
|
||||
See documentation in topics/media-pipeline.rst
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
|
|
|
|||
Loading…
Reference in New Issue