From dce48b8654977d3f7a003addd507b0aa5e71a76c Mon Sep 17 00:00:00 2001 From: Elias Dorneles Date: Sat, 11 Apr 2015 13:57:55 -0300 Subject: [PATCH 1/3] documenting Files Pipeline together with Images Pipeline --- docs/index.rst | 2 +- docs/topics/images.rst | 250 ++++++++++++++++++++++++++--------------- 2 files changed, 159 insertions(+), 93 deletions(-) diff --git a/docs/index.rst b/docs/index.rst index de3e015d5..f61e7f348 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -181,7 +181,7 @@ Solving specific problems Learn how to find and get rid of memory leaks in your crawler. :doc:`topics/images` - Download static images associated with your scraped items. + Download files and/or images associated with your scraped items. :doc:`topics/ubuntu` Install latest Scrapy packages easily on Ubuntu diff --git a/docs/topics/images.rst b/docs/topics/images.rst index 890c7fd4a..ab4a98ebd 100644 --- a/docs/topics/images.rst +++ b/docs/topics/images.rst @@ -1,76 +1,94 @@ .. _topics-images: -======================= -Downloading Item Images -======================= +=========================================== +Downloading and processing files and images +=========================================== .. currentmodule:: scrapy.contrib.pipeline.images -Scrapy provides an :doc:`item pipeline ` for downloading -images attached to a particular item, for example, when you scrape products and -also want to download their images locally. +Scrapy provides reusable :doc:`item pipelines ` for +downloading fies attached to a particular item (for example, when you scrape +products and also want to download their images locally). These pipelines share +a bit of functionality and structure (we refer to them as media pipelines), but +typically you'll either use the Files Pipeline or the Images Pipeline. -This pipeline, called the Images Pipeline and implemented in the -:class:`ImagesPipeline` class, provides a convenient way for -downloading and storing images locally with some additional features: +Both pipelines implement these features: + +* Avoid re-downloading media that was downloaded recently +* Specifying where to store the files (filesystem directory, Amazon S3 bucket) + +The Images Pipeline has a few extra functions for processing images: * Convert all downloaded images to a common format (JPG) and mode (RGB) -* Avoid re-downloading images which were downloaded recently * Thumbnail generation * Check images width/height to make sure they meet a minimum constraint -This pipeline also keeps an internal queue of those images which are currently -being scheduled for download, and connects those items that arrive containing -the same image, to that queue. This avoids downloading the same image more than +The pipelines also keep an internal queue of those images which are currently +being scheduled for download, and connect those items that arrive containing +the same image to that queue. This avoids downloading the same media more than once when it's shared by several items. -`Pillow`_ is used for thumbnailing and normalizing images to JPEG/RGB format, -so you need to install this library in order to use the images pipeline. -`Python Imaging Library`_ (PIL) should also work in most cases, but it -is known to cause troubles in some setups, so we recommend to use `Pillow`_ -instead of `PIL `_. +The Images Pipeline uses `Pillow`_ for thumbnailing and normalizing images to +JPEG/RGB format, so you need to install this library in order to use it. +`Python Imaging Library`_ (PIL) should also work in most cases, but it is known +to cause troubles in some setups, so we recommend to use `Pillow`_ instead of +`PIL `_. .. _Pillow: https://github.com/python-pillow/Pillow .. _Python Imaging Library: http://www.pythonware.com/products/pil/ -Using the Images Pipeline +Using the Files Pipeline ========================= -The typical workflow, when using the :class:`ImagesPipeline` goes like +The typical workflow, when using the :class:`FilesPipeline` goes like this: -1. In a Spider, you scrape an item and put the URLs of its images into a - ``image_urls`` field. +1. In a Spider, you scrape an item and put the URLs of the desired into a + ``file_urls`` field. 2. The item is returned from the spider and goes to the item pipeline. -3. When the item reaches the :class:`ImagesPipeline`, the URLs in the - ``image_urls`` field are scheduled for download using the standard +3. When the item reaches the :class:`FilesPipeline`, the URLs in the + ``file_urls`` field are scheduled for download using the standard Scrapy scheduler and downloader (which means the scheduler and downloader middlewares are reused), but with a higher priority, processing them before other pages are scraped. The item remains "locked" at that particular pipeline stage - until the images have finish downloading (or fail for some reason). + until the files have finish downloading (or fail for some reason). -4. When the images are downloaded another field (``images``) will be populated +4. When the files are downloaded, another field (``files``) will be populated with the results. This field will contain a list of dicts with information - about the images downloaded, such as the downloaded path, the original - scraped url (taken from the ``image_urls`` field) , and the image checksum. - The images in the list of the ``images`` field will retain the same order of - the original ``image_urls`` field. If some image failed downloading, an - error will be logged and the image won't be present in the ``images`` field. + about the downloaded files, such as the downloaded path, the original + scraped url (taken from the ``file_urls`` field) , and the file checksum. + The files in the list of the ``files`` field will retain the same order of + the original ``file_urls`` field. If some file failed downloading, an + error will be logged and the file won't be present in the ``files`` field. + + +Using the Images Pipeline +========================= + +Using the :class:`ImagesPipeline` is a lot like using the :class:`FilesPipeline`, +except the default field names used are different: you use ``image_urls`` for +the image URLs of an item and it will populate an ``images`` field for the information +about the downloaded images. + +The advantage of using the :class:`ImagesPipeline` for image files is that you +can configure some extra functions like generating thumbnails and filtering +the images based on their size. Usage example ============= -In order to use the image pipeline first -:ref:`enable it `. +In order to use a media pipeline first, :ref:`enable it +`. -Then, if a spider returns a dict with 'image_urls' key, -the pipeline will put the results under 'images' key. +Then, if a spider returns a dict with the URLs key ('file_urls' or +'image_urls', for the Files or Images Pipeline respectively), the pipeline will +put the results under respective key ('files' or images'). -If you prefer to use :class:`~.Item` then define a custom -item with the ``image_urls`` and ``images`` fields:: +If you prefer to use :class:`~.Item`, then define a custom item with the +necessary fields, like in this example for Images Pipeline:: import scrapy @@ -80,42 +98,57 @@ item with the ``image_urls`` and ``images`` fields:: image_urls = scrapy.Field() images = scrapy.Field() -If you need something more complex and want to override the custom images -pipeline behaviour, see :ref:`topics-images-override`. +If you need something more complex and want to override the custom pipeline +behaviour, see :ref:`topics-media-pipeline-override`. -.. _topics-images-enabling: +.. _topics-media-pipeline-enabling: -Enabling your Images Pipeline +Enabling your Media Pipeline ============================= .. setting:: IMAGES_STORE +.. setting:: FILES_STORE -To enable your images pipeline you must first add it to your project -:setting:`ITEM_PIPELINES` setting:: +To enable your media pipeline you must first add it to your project +:setting:`ITEM_PIPELINES` setting. + +For Images Pipeline, use:: ITEM_PIPELINES = {'scrapy.contrib.pipeline.images.ImagesPipeline': 1} -And set the :setting:`IMAGES_STORE` setting to a valid directory that will be -used for storing the downloaded images. Otherwise the pipeline will remain -disabled, even if you include it in the :setting:`ITEM_PIPELINES` setting. +For Files Pipeline, use:: -For example:: + ITEM_PIPELINES = {'scrapy.contrib.pipeline.files.FilesPipeline': 1} + + +.. note:: + You can also use both the Files and Images Pipeline at the same time. + + +Then, configure the target storage setting to a valid value that will be used +for storing the downloaded images. Otherwise the pipeline will remain disabled, +even if you include it in the :setting:`ITEM_PIPELINES` setting. + +For the Files Pipeline, set the :setting:`FILES_STORE` setting:: + + FILES_STORE = '/path/to/valid/dir' + +For the Images Pipeline, set the :setting:`IMAGES_STORE` setting:: IMAGES_STORE = '/path/to/valid/dir' -Images Storage -============== +Supported Storage +================= File system is currently the only officially supported storage, but there is -also (undocumented) support for `Amazon S3`_. +also (undocumented) support for storing files in `Amazon S3`_. .. _Amazon S3: http://aws.amazon.com/s3/ File system storage ------------------- -The images are stored in files (one per image), using a `SHA1 hash`_ of their -URLs for the file names. +The files are stored using a `SHA1 hash`_ of their URLs for the file names. For example, the following image URL:: @@ -132,29 +165,36 @@ Will be downloaded and stored in the following file:: Where: * ```` is the directory defined in :setting:`IMAGES_STORE` setting +for the Images Pipeline. * ``full`` is a sub-directory to separate full images from thumbnails (if used). For more info see :ref:`topics-images-thumbnails`. + Additional features =================== -Image expiration +File expiration ---------------- .. setting:: IMAGES_EXPIRES +.. setting:: FILES_EXPIRES -The Image Pipeline avoids downloading images that were downloaded recently. To -adjust this retention delay use the :setting:`IMAGES_EXPIRES` setting, which +The Image Pipeline avoids downloading files that were downloaded recently. To +adjust this retention delay use the :setting:`FILES_EXPIRES` setting (or +:setting:`IMAGES_EXPIRES`, in case of Images Pipeline), which specifies the delay in number of days:: - # 90 days of delay for image expiration - IMAGES_EXPIRES = 90 + # 90 days of delay for files expiration + FILES_EXPIRES = 90 + + # 30 days of delay for images expiration + IMAGES_EXPIRES = 30 .. _topics-images-thumbnails: -Thumbnail generation --------------------- +Thumbnail generation for images +------------------------------- The Images Pipeline can automatically create thumbnails of the downloaded images. @@ -200,8 +240,9 @@ Filtering out small images .. setting:: IMAGES_MIN_WIDTH -You can drop images which are too small, by specifying the minimum allowed size -in the :setting:`IMAGES_MIN_HEIGHT` and :setting:`IMAGES_MIN_WIDTH` settings. +When using the Images Pipeline, you can drop images which are too small, by +specifying the minimum allowed size in the :setting:`IMAGES_MIN_HEIGHT` and +:setting:`IMAGES_MIN_WIDTH` settings. For example:: @@ -212,45 +253,45 @@ Note: these size constraints don't affect thumbnail generation at all. By default, there are no size constraints, so all images are processed. -.. _topics-images-override: +.. _topics-media-pipeline-override: -Implementing your custom Images Pipeline -======================================== +Extending the Media Pipelines +============================= -.. module:: scrapy.contrib.pipeline.images - :synopsis: Images Pipeline +.. module:: scrapy.contrib.pipeline.files + :synopsis: Files Pipeline -Here are the methods that you should override in your custom Images Pipeline: +See here the methods that you can override in your custom Files Pipeline: -.. class:: ImagesPipeline +.. class:: FilesPipeline - .. method:: get_media_requests(item, info) + .. method:: FilesPipeline.get_media_requests(item, info) As seen on the workflow, the pipeline will get the URLs of the images to - download from the item. In order to do this, you must override the + download from the item. In order to do this, you can override the :meth:`~get_media_requests` method and return a Request for each - image URL:: + file URL:: def get_media_requests(self, item, info): - for image_url in item['image_urls']: - yield scrapy.Request(image_url) + for file_url in item['file_urls']: + yield scrapy.Request(file_url) Those requests will be processed by the pipeline and, when they have finished downloading, the results will be sent to the :meth:`~item_completed` method, as a list of 2-element tuples. - Each tuple will contain ``(success, image_info_or_failure)`` where: + Each tuple will contain ``(success, file_info_or_error)`` where: * ``success`` is a boolean which is ``True`` if the image was downloaded successfully or ``False`` if it failed for some reason - * ``image_info_or_error`` is a dict containing the following keys (if success + * ``file_info_or_error`` is a dict containing the following keys (if success is ``True``) or a `Twisted Failure`_ if there was a problem. - * ``url`` - the url where the image was downloaded from. This is the url of + * ``url`` - the url where the file was downloaded from. This is the url of the request returned from the :meth:`~get_media_requests` method. - * ``path`` - the path (relative to :setting:`IMAGES_STORE`) where the image + * ``path`` - the path (relative to :setting:`FILES_STORE`) where the file was stored * ``checksum`` - a `MD5 hash`_ of the image contents @@ -263,21 +304,17 @@ Here are the methods that you should override in your custom Images Pipeline: [(True, {'checksum': '2b00042f7481c7b056c4b410d28f33cf', - 'path': 'full/7d97e98f8af710c7e7fe703abc8f639e0ee507c4.jpg', - 'url': 'http://www.example.com/images/product1.jpg'}), - (True, - {'checksum': 'b9628c4ab9b595f72f280b90c4fd093d', - 'path': 'full/1ca5879492b8fd606df1964ea3c1e2f4520f076f.jpg', - 'url': 'http://www.example.com/images/product2.jpg'}), + 'path': 'full/0a79c461a4062ac383dc4fade7bc09f1384a3910.jpg', + 'url': 'http://www.example.com/files/product1.pdf'}), (False, Failure(...))] By default the :meth:`get_media_requests` method returns ``None`` which - means there are no images to download for the item. + means there are no files to download for the item. - .. method:: item_completed(results, items, info) + .. method:: FilesPipeline.item_completed(results, items, info) - The :meth:`ImagesPipeline.item_completed` method called when all image + The :meth:`FilesPipeline.item_completed` method called when all file requests for a single item have completed (either finished downloading, or failed for some reason). @@ -286,21 +323,50 @@ Here are the methods that you should override in your custom Images Pipeline: return (or drop) the item, as you would in any pipeline. Here is an example of the :meth:`~item_completed` method where we - store the downloaded image paths (passed in results) in the ``image_paths`` - item field, and we drop the item if it doesn't contain any images:: + store the downloaded file paths (passed in results) in the ``file_paths`` + item field, and we drop the item if it doesn't contain any files:: from scrapy.exceptions import DropItem def item_completed(self, results, item, info): - image_paths = [x['path'] for ok, x in results if ok] - if not image_paths: - raise DropItem("Item contains no images") - item['image_paths'] = image_paths + file_paths = [x['path'] for ok, x in results if ok] + if not file_paths: + raise DropItem("Item contains no files") + item['file_paths'] = file_paths return item By default, the :meth:`item_completed` method returns the item. +.. module:: scrapy.contrib.pipeline.images + :synopsis: Images Pipeline + +See here the methods that you can override in your custom Images Pipeline: + +.. class:: ImagesPipeline + + The :class:`ImagesPipeline` is an extension of the :class:`FilesPipeline`, + customizing the field names and adding custom behavior for images. + + .. method:: ImagesPipeline.get_media_requests(item, info) + + Works the same way as :meth:`FilesPipeline.get_media_requests` method, + but using a different field name for image urls. + + Must return a Request for each image URL. + + .. method:: ImagesPipeline.item_completed(results, items, info) + + The :meth:`ImagesPipeline.item_completed` method is called when all image + requests for a single item have completed (either finished downloading, or + failed for some reason). + + Works the same way as :meth:`FilesPipeline.item_completed` method, + but using a different field names for storing image downloading results. + + By default, the :meth:`item_completed` method returns the item. + + Custom Images pipeline example ============================== From fd1c688ad637417db539463e69ef0d59d9c2e61e Mon Sep 17 00:00:00 2001 From: Elias Dorneles Date: Wed, 15 Apr 2015 14:21:48 -0300 Subject: [PATCH 2/3] addressing issues from review at #1150 --- docs/topics/images.rst | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/docs/topics/images.rst b/docs/topics/images.rst index ab4a98ebd..903359745 100644 --- a/docs/topics/images.rst +++ b/docs/topics/images.rst @@ -15,7 +15,7 @@ typically you'll either use the Files Pipeline or the Images Pipeline. Both pipelines implement these features: * Avoid re-downloading media that was downloaded recently -* Specifying where to store the files (filesystem directory, Amazon S3 bucket) +* Specifying where to store the media (filesystem directory, Amazon S3 bucket) The Images Pipeline has a few extra functions for processing images: @@ -23,20 +23,11 @@ The Images Pipeline has a few extra functions for processing images: * Thumbnail generation * Check images width/height to make sure they meet a minimum constraint -The pipelines also keep an internal queue of those images which are currently -being scheduled for download, and connect those items that arrive containing -the same image to that queue. This avoids downloading the same media more than +The pipelines also keep an internal queue of those media URLs which are currently +being scheduled for download, and connect those responses that arrive containing +the same media to that queue. This avoids downloading the same media more than once when it's shared by several items. -The Images Pipeline uses `Pillow`_ for thumbnailing and normalizing images to -JPEG/RGB format, so you need to install this library in order to use it. -`Python Imaging Library`_ (PIL) should also work in most cases, but it is known -to cause troubles in some setups, so we recommend to use `Pillow`_ instead of -`PIL `_. - -.. _Pillow: https://github.com/python-pillow/Pillow -.. _Python Imaging Library: http://www.pythonware.com/products/pil/ - Using the Files Pipeline ========================= @@ -76,6 +67,15 @@ The advantage of using the :class:`ImagesPipeline` for image files is that you can configure some extra functions like generating thumbnails and filtering the images based on their size. +The Images Pipeline uses `Pillow`_ for thumbnailing and normalizing images to +JPEG/RGB format, so you need to install this library in order to use it. +`Python Imaging Library`_ (PIL) should also work in most cases, but it is known +to cause troubles in some setups, so we recommend to use `Pillow`_ instead of +`PIL `_. + +.. _Pillow: https://github.com/python-pillow/Pillow +.. _Python Imaging Library: http://www.pythonware.com/products/pil/ + Usage example ============= From a1f3b3c7f05c505fa9f0b907681fc97f7548cfae Mon Sep 17 00:00:00 2001 From: Elias Dorneles Date: Wed, 15 Apr 2015 14:26:08 -0300 Subject: [PATCH 3/3] images.rst -> media-pipeline.rst --- docs/index.rst | 4 ++-- docs/topics/{images.rst => media-pipeline.rst} | 2 +- scrapy/contrib/pipeline/files.py | 2 ++ scrapy/contrib/pipeline/images.py | 2 +- 4 files changed, 6 insertions(+), 4 deletions(-) rename docs/topics/{images.rst => media-pipeline.rst} (99%) diff --git a/docs/index.rst b/docs/index.rst index f61e7f348..3b245560c 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -148,7 +148,7 @@ Solving specific problems topics/firefox topics/firebug topics/leaks - topics/images + topics/media-pipeline topics/ubuntu topics/scrapyd topics/autothrottle @@ -180,7 +180,7 @@ Solving specific problems :doc:`topics/leaks` Learn how to find and get rid of memory leaks in your crawler. -:doc:`topics/images` +:doc:`topics/media-pipeline` Download files and/or images associated with your scraped items. :doc:`topics/ubuntu` diff --git a/docs/topics/images.rst b/docs/topics/media-pipeline.rst similarity index 99% rename from docs/topics/images.rst rename to docs/topics/media-pipeline.rst index 903359745..4b88c60fd 100644 --- a/docs/topics/images.rst +++ b/docs/topics/media-pipeline.rst @@ -1,4 +1,4 @@ -.. _topics-images: +.. _topics-media-pipeline: =========================================== Downloading and processing files and images diff --git a/scrapy/contrib/pipeline/files.py b/scrapy/contrib/pipeline/files.py index 9e803aca0..608614865 100644 --- a/scrapy/contrib/pipeline/files.py +++ b/scrapy/contrib/pipeline/files.py @@ -1,5 +1,7 @@ """ Files Pipeline + +See documentation in topics/media-pipeline.rst """ import hashlib diff --git a/scrapy/contrib/pipeline/images.py b/scrapy/contrib/pipeline/images.py index b12995f09..3cbfed49b 100644 --- a/scrapy/contrib/pipeline/images.py +++ b/scrapy/contrib/pipeline/images.py @@ -1,7 +1,7 @@ """ Images Pipeline -See documentation in topics/images.rst +See documentation in topics/media-pipeline.rst """ import hashlib