From 2f8c052484c51ba63037fa7256c027e4376fe3bc Mon Sep 17 00:00:00 2001 From: Lucian Ursu Date: Sun, 18 Apr 2010 23:39:54 -0300 Subject: [PATCH] #154: Language fixes to the documentation --- docs/topics/architecture.rst | 10 ++-- docs/topics/downloader-middleware.rst | 56 +++++++++++------------ docs/topics/email.rst | 66 +++++++++++++++++---------- docs/topics/exceptions.rst | 4 +- docs/topics/exporters.rst | 12 ++--- docs/topics/extensions.rst | 34 +++++++------- docs/topics/firebug.rst | 10 ++-- docs/topics/firefox.rst | 8 ++-- docs/topics/images.rst | 26 +++++------ 9 files changed, 122 insertions(+), 104 deletions(-) diff --git a/docs/topics/architecture.rst b/docs/topics/architecture.rst index af1eb5fac..fd24e6c88 100644 --- a/docs/topics/architecture.rst +++ b/docs/topics/architecture.rst @@ -4,7 +4,7 @@ Architecture overview ===================== -This document describes the architecture of Scrapy and how their components +This document describes the architecture of Scrapy and how its components interact. Overview @@ -41,12 +41,12 @@ Downloader ---------- The Downloader is responsible for fetching web pages and feeding them to the -engine which, in turns, feeds them to the spiders. +engine which, in turn, feeds them to the spiders. Spiders ------- -Spiders are custom classes written by Scrapy users to parse response and +Spiders are custom classes written by Scrapy users to parse responses and extract items (aka scraped items) from them or additional URLs (requests) to follow. Each spider is able to handle a specific domain (or group of domains). For more information see :ref:`topics-spiders`. @@ -64,7 +64,7 @@ Downloader middlewares Downloader middlewares are specific hooks that sit between the Engine and the Downloader and process requests when they pass from the Engine to the -downloader, and responses that pass from Downloader to the Engine. They provide +Downloader, and responses that pass from Downloader to the Engine. They provide a convenient mechanism for extending Scrapy functionality by plugging custom code. For more information see :ref:`topics-downloader-middleware`. @@ -80,7 +80,7 @@ functionality by plugging custom code. For more information see Scheduler middlewares --------------------- -Spider middlewares are specific hooks that sit between the Engine and the +Scheduler middlewares are specific hooks that sit between the Engine and the Scheduler and process requests when they pass from the Engine to the Scheduler and vice-versa. They provide a convenient mechanism for extending Scrapy functionality by plugging custom code. diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst index 15ead63be..ec3679649 100644 --- a/docs/topics/downloader-middleware.rst +++ b/docs/topics/downloader-middleware.rst @@ -35,9 +35,9 @@ where you want to insert the middleware. The order does matter because each middleware performs a different action and your middleware could depend on some previous (or subsequent) middleware being applied. -If you want to disable a builtin middleware (the ones defined in +If you want to disable a built-in middleware (the ones defined in :setting:`DOWNLOADER_MIDDLEWARES_BASE` and enabled by default) you must define it -in your project :setting:`DOWNLOADER_MIDDLEWARES` setting and assign `None` +in your project's :setting:`DOWNLOADER_MIDDLEWARES` setting and assign `None` as its value. For example, if you want to disable the off-site middleware:: DOWNLOADER_MIDDLEWARES = { @@ -67,24 +67,24 @@ single Python class that defines one or more of the following methods: :class:`~scrapy.http.Response` object, or a :class:`~scrapy.http.Request` object. - If returns ``None``, Scrapy will continue processing this request, executing all + If it returns ``None``, Scrapy will continue processing this request, executing all other middlewares until, finally, the appropriate downloader handler is called the request performed (and its response downloaded). - If returns a :class:`~scrapy.http.Response` object, Scrapy won't bother + If it returns a :class:`~scrapy.http.Response` object, Scrapy won't bother calling ANY other request or exception middleware, or the appropriate download function; it'll return that Response. Response middleware is - always called on every response. + always called on every Response. - If returns a :class:`~scrapy.http.Request` object, the returned request will be - re-scheduled (in the Scheduler) to be downloaded in the future. The callback of + If it returns a :class:`~scrapy.http.Request` object, the returned request will be + rescheduled (in the Scheduler) to be downloaded in the future. The callback of the original request will always be called. If the new request has a callback it will be called with the response downloaded, and the output of that callback will then be passed to the original callback. If the new request doesn't have a callback, the response downloaded will be just passed to the original request callback. - If returns an :exc:`~scrapy.core.exceptions.IgnoreRequest` exception, the + If it returns an :exc:`~scrapy.core.exceptions.IgnoreRequest` exception, the entire request will be dropped completely and its callback never called. :param request: the request being processed @@ -95,14 +95,14 @@ single Python class that defines one or more of the following methods: .. method:: process_response(request, response, spider) - meth:`process_response` should return a :class:`~scrapy.http.Response` + :meth:`process_response` should return a :class:`~scrapy.http.Response` object or raise a :exc:`~scrapy.core.exceptions.IgnoreRequest` exception. - If returns a :class:`~scrapy.http.Response` (it could be the same given - response, or a brand-new one) that response will continue to be processed + If it returns a :class:`~scrapy.http.Response` (it could be the same given + response, or a brand-new one), that response will continue to be processed with the :meth:`process_response` of the next middleware in the pipeline. - If returns an :exc:`~scrapy.core.exceptions.IgnoreRequest` exception, the + If it returns an :exc:`~scrapy.core.exceptions.IgnoreRequest` exception, the response will be dropped completely and its callback never called. :param request: the request that originated the response @@ -130,17 +130,17 @@ single Python class that defines one or more of the following methods: If it returns a :class:`~scrapy.http.Response` object, the response middleware kicks in, and won't bother calling any other exception middleware. - If it returns a :class:`~scrapy.http.Request` object, returned request is - used to instruct a immediate redirection. Redirection is handled inside - middleware scope, and the original request won't finish until redirected - request is completed. This stop :meth:`process_download_exception` - middleware as returning Response would do. + If it returns a :class:`~scrapy.http.Request` object, the returned request is + used to instruct an immediate redirection. + The original request won't finish until the redirected + request is completed. This stops the :meth:`process_download_exception` + middleware the same as returning Response would do. :param request: the request that generated the exception :type request: is a :class:`~scrapy.http.Request` object :param exception: the raised exception - :type exception: a ``Exception`` object + :type exception: an ``Exception`` object :param spider: the spider for which this request is intended :type spider: :class:`~scrapy.spider.BaseSpider` object @@ -190,7 +190,7 @@ HttpAuthMiddleware This middleware authenticates all requests generated from certain spiders using `Basic access authentication`_ (aka. HTTP auth). - To enable HTTP authentication from certain spiders set the ``http_user`` + To enable HTTP authentication from certain spiders, set the ``http_user`` and ``http_pass`` attributes of those spiders. Example:: @@ -219,7 +219,7 @@ HttpCacheMiddleware anything from the Internet. The HTTP cache is useful for testing spiders faster (without having to wait for - downloads every time) and for trying your spider off-line when you don't have + downloads every time) and for trying your spider offline, when you don't have an Internet connection. File system storage @@ -227,7 +227,7 @@ File system storage By default, the :class:`HttpCacheMiddleware` uses a file system storage with the following structure: -Each request/response pair is stored in a different directory containing with +Each request/response pair is stored in a different directory containing the following files: * ``request_body`` - the plain request body @@ -235,7 +235,7 @@ the following files: * ``response_body`` - the plain response body * ``response_headers`` - the request headers (in raw HTTP format) * ``meta`` - some metadata of this cache resource in Python ``repr()`` format - (for easy grepeability) + (grep-friendly format) * ``pickled_meta`` - the same metadata in ``meta`` but pickled for more efficient deserialization @@ -262,7 +262,7 @@ HTTPCACHE_DIR Default: ``''`` (empty string) -The directory to use for storing the (low-level) HTTP cache. If empty the HTTP +The directory to use for storing the (low-level) HTTP cache. If empty, the HTTP cache will be disabled. .. setting:: HTTPCACHE_EXPIRATION_SECS @@ -274,7 +274,7 @@ Default: ``0`` Number of seconds to use for HTTP cache expiration. Requests that were cached before this time will be re-downloaded. If zero, cached requests will always -expire. Negative numbers means requests will never expire. +expire. A negative number means requests will never expire. .. setting:: HTTPCACHE_IGNORE_MISSING @@ -319,7 +319,7 @@ HttpProxyMiddleware This middleware sets the HTTP proxy to use for requests, by setting the ``proxy`` meta value to :class:`~scrapy.http.Request` objects. - Like the Python standard library modules `urllib`_ and `urllib2`_ it obeys + Like the Python standard library modules `urllib`_ and `urllib2`_, it obeys the following enviroment variables: * ``http_proxy`` @@ -360,7 +360,7 @@ RetryMiddleware Failed pages are collected on the scraping process and rescheduled at the end, once the spider has finished crawling all regular (non failed) pages. - Once there is no more failed pages to retry this middleware sends a signal + Once there are no more failed pages to retry, this middleware sends a signal (retry_complete), so other extensions could connect to that signal. The :class:`RetryMiddleware` can be configured through the following @@ -389,7 +389,7 @@ RobotsTxtMiddleware standard. To make sure Scrapy respects robots.txt make sure the middleware is enabled - amd the :setting:`ROBOTSTXT_OBEY` setting is enabled. + and the :setting:`ROBOTSTXT_OBEY` setting is enabled. .. warning:: Keep in mind that, if you crawl using multiple concurrent requests per domain, Scrapy could still download some forbidden pages @@ -405,7 +405,7 @@ DownloaderStats .. class:: DownloaderStats - Middleware that store stats of all requests, responses and exceptions that + Middleware that stores stats of all requests, responses and exceptions that pass through it. To use this middleware you must enable the :setting:`DOWNLOADER_STATS` diff --git a/docs/topics/email.rst b/docs/topics/email.rst index 7fc2f9d60..a2c7568df 100644 --- a/docs/topics/email.rst +++ b/docs/topics/email.rst @@ -1,25 +1,28 @@ .. _topics-email: ============= -Sending email +Sending e-mail ============= .. module:: scrapy.mail - :synopsis: Helpers to easily send e-mail. + :synopsis: Email sending facility -Although Python makes sending e-mail relatively easy via the `smtplib`_ -library, Scrapy provides its own class for sending emails which is very easy to -use and it's implemented using `Twisted non-blocking IO`_, to avoid affecting -the crawling performance. +Although Python makes sending e-mails relatively easy via the `smtplib`_ +library, Scrapy provides its own facility for sending e-mails which is very easy +to use and it's implemented using `Twisted non-blocking IO`_, to avoid +interfering with the non-blocking IO of the crawler. + +It's also very easy to configure, having only a few settings. .. _smtplib: http://docs.python.org/library/smtplib.html +.. _Twisted non-blocking IO: http://twistedmatrix.com/projects/core/documentation/howto/async.html It also has built-in support for sending attachments. Quick example ============= -Here's a quick example of how to send an email (without attachments):: +Here's a quick example of how to send an e-mail (without attachments):: from scrapy.mail import MailSender @@ -34,30 +37,45 @@ uses `Twisted non-blocking IO`_, like the rest of the framework. .. class:: MailSender(smtphost, mailfrom) - ``smtphost`` is a string with the SMTP host to use for sending the emails. - If omitted, :setting:`MAIL_HOST` will be used. + :param smtphost: the SMTP host to use for sending the emails. If omitted, the + :setting:`MAIL_HOST` setting will be used. + :type smtphost: str - ``mailfrom`` is a string with the email address to use for sending messages - (in the ``From:`` header). If omitted, :setting:`MAIL_FROM` will be used. + :param mailfrom: the address used to send emails (in the ``From:`` header). + If omitted, the :setting:`MAIL_FROM` setting will be used. + :type mailfrom: str -.. method:: MailSender.send(to, subject, body, cc=None, attachs=()) + .. method:: send(to, subject, body, cc=None, attachs=()) - Send mail to the given recipients + Send email to the given recipients - ``to`` is a list of email recipients + :param to: the e-mail recipients + :type to: list - ``subject`` is a string with the subject of the message + :param subject: the subject of the e-mail + :type subject: str - ``cc`` is a list of emails to CC + :param cc: the e-mails to CC + :type cc: list - ``body`` is a string with the body of the message + :param body: the e-mail body + :type body: str - ``attachs`` is an iterable of tuples (attach_name, mimetype, file_object) - where: - - ``attach_name`` is a string with the name will appear on the emails attachment - ``mimetype`` is the mimetype of the attachment - ``file_object`` is a readable file object + :param attachs: an iterable of tuples ``(attach_name, mimetype, + file_object)`` where ``attach_name`` is a string with the name that will + appear on the e-mail's attachment, ``mimetype`` is the mimetype of the + attachment and ``file_object`` is a readable file object with the + contents of the attachment + :type attachs: iterable -.. _Twisted non-blocking IO: http://twistedmatrix.com/projects/core/documentation/howto/async.html +MailSender settings +=================== + +These settings define the default constructor values of the :class:`MailSender` +class, and can be used to configure e-mail notifications in your project without +writing any code (for those extensions that use the :class:`MailSender` class): + +* :setting:`MAIL_FROM` +* :setting:`MAIL_HOST` + diff --git a/docs/topics/exceptions.rst b/docs/topics/exceptions.rst index 8c9ab7594..eb7f7686b 100644 --- a/docs/topics/exceptions.rst +++ b/docs/topics/exceptions.rst @@ -27,7 +27,7 @@ IgnoreRequest .. exception:: IgnoreRequest -This exception can be raised by the Scheduler or any downlaoder middleware to +This exception can be raised by the Scheduler or any downloader middleware to indicate that the request should be ignored. NotConfigured @@ -36,7 +36,7 @@ NotConfigured .. exception:: NotConfigured This exception can be raised by some components to indicate that they will -remain disabled. Those component include: +remain disabled. Those components include: * Extensions * Item pipelines diff --git a/docs/topics/exporters.rst b/docs/topics/exporters.rst index 711df19ce..e8f6bef75 100644 --- a/docs/topics/exporters.rst +++ b/docs/topics/exporters.rst @@ -20,10 +20,10 @@ Using Item Exporters If you are in a hurry, and just want to use an Item Exporter as an :doc:`Item Pipeline ` see the :ref:`File Export Pipeline `. Otherwise, if you want to know how Item Exporters -work, or need more custom functionality (not covered by the :ref:`File Export -Pipeline `) continue reading below. +work or need more custom functionality (not covered by the :ref:`File Export +Pipeline `), continue reading below. -In order to use a Item Exporter, you must instantiate it with its required +In order to use an Item Exporter, you must instantiate it with its required args. Each Item Exporter requires different arguments, so check each exporter documentation to be sure, in :ref:`topics-exporters-reference`. After you have instantiated you exporter, you have to: @@ -72,7 +72,7 @@ Exporter to export scraped items to different files, one per spider:: Serialization of item fields ============================ -By default the field values are passed unmodified to the underlying +By default, the field values are passed unmodified to the underlying serialization library, and the decision of how to serialize them is delegated to each particular serialization library. @@ -119,7 +119,7 @@ Example:: class ProductXmlExporter(XmlItemExporter): def serialize_field(self, field, name, value): - if filed == 'price': + if field == 'price': return '$ %s' % str(value) return super(Product, self).serialize_field(field, name, value) @@ -241,7 +241,7 @@ XmlItemExporter - Unless overriden in :meth:`serialize_field` method, multi-valued fields are + Unless overriden in the :meth:`serialize_field` method, multi-valued fields are exported by serializing each value inside a ```` element. This is for convenience, as multi-valued fields are very common. diff --git a/docs/topics/extensions.rst b/docs/topics/extensions.rst index 1bde87bb7..fc7367bac 100644 --- a/docs/topics/extensions.rst +++ b/docs/topics/extensions.rst @@ -4,7 +4,7 @@ Extensions ========== -The extensions framework provide a mechanism for inserting your own +The extensions framework provides a mechanism for inserting your own custom functionality into Scrapy. Extensions are just regular classes that are instantiated at Scrapy startup, @@ -50,8 +50,8 @@ orders though, and they are typically irrelevant, ie. it doesn't matter in which order the extensions are loaded because they don't depend on each other [1]. -However this feature can be exploited if you need to add an extension which -depends on other extension already loaded. +However, this feature can be exploited if you need to add an extension which +depends on other extensions already loaded. [1] This is is why the :setting:`EXTENSIONS_BASE` setting in Scrapy (which contains all built-in extensions enabled by default) defines all the extensions @@ -63,7 +63,7 @@ Available, enabled and disabled extensions Not all available extensions will be enabled. Some of them usually depend on a particular setting. For example, the HTTP Cache extension is available by default but disabled unless the :setting:`HTTPCACHE_DIR` setting is set. Both enabled -and disabled extension can be accessed through the +and disabled extensions can be accessed through the :ref:`topics-extensions-ref-manager`. Accessing enabled extensions @@ -76,9 +76,9 @@ loaded. For example, to access the ``WebConsole`` extension:: from scrapy.extension import extensions webconsole_extension = extensions.enabled['WebConsole'] -.. seealso:: +.. see also:: - :ref:`topics-extensions-ref-manager`, for the complete Extension manager + :ref:`topics-extensions-ref-manager`, for the complete Extension Manager reference. Writing your own extension @@ -93,7 +93,7 @@ All extension initialization code must be performed in the class constructor disabled. Otherwise, the extension will be enabled. Let's take a look at the following example extension which just logs a message -everytime a domain/spider is opened and closed:: +every time a domain/spider is opened and closed:: from scrapy.xlib.pydispatch import dispatcher from scrapy.core import signals @@ -113,7 +113,7 @@ everytime a domain/spider is opened and closed:: .. _topics-extensions-ref-manager: -Extension manager +Extension Manager ================= .. module:: scrapy.extension @@ -127,7 +127,7 @@ how you :ref:`configure the downloader middlewares .. class:: ExtensionManager - The extension manager is a singleton object, which is instantiated at module + The Extension Manager is a singleton object, which is instantiated at module loading time and can be accessed like this:: from scrapy.extension import extensions @@ -190,7 +190,7 @@ Core Stats extension .. class:: CoreStats -Enable the collection of core statistics, provided the stats collection are +Enable the collection of core statistics, provided the stats collection is enabled (see :ref:`topics-stats`). .. _topics-extensions-ref-webconsole: @@ -246,10 +246,10 @@ Memory usage extension Allows monitoring the memory used by a Scrapy process and: -1, send a notification email when it exceeds a certain value +1, send a notification e-mail when it exceeds a certain value 2. terminate the Scrapy process when it exceeds a certain value -The notification emails can be triggered when a certain warning value is +The notification e-mails can be triggered when a certain warning value is reached (:setting:`MEMUSAGE_WARNING_MB`) and when the maximum value is reached (:setting:`MEMUSAGE_LIMIT_MB`) which will also cause the Scrapy process to be terminated. @@ -271,9 +271,9 @@ Memory debugger extension .. class:: scrapy.contrib.memdebug.MemoryDebugger A memory debugger which collects some info about objects uncollected by the -garbage collector and libxml2 memory leaks. To enable this extension turn on +garbage collector and libxml2 memory leaks. To enable this extension, turn on the :setting:`MEMDEBUG_ENABLED` setting. The report will be printed to standard -output. If the :setting:`MEMDEBUG_NOTIFY` setting contains a list of emails the +output. If the :setting:`MEMDEBUG_NOTIFY` setting contains a list of e-mails the report will also be sent to those addresses. Close spider extension @@ -299,7 +299,7 @@ Default: ``0`` An integer which specifies a number of seconds. If the spider remains open for more than that number of second, it will be automatically closed with the -reason ``closespider_timeout``. If zero (or non set) spiders won't be closed by +reason ``closespider_timeout``. If zero (or non set), spiders won't be closed by timeout. .. setting:: CLOSESPIDER_ITEMPASSED @@ -312,7 +312,7 @@ Default: ``0`` An integer which specifies a number of items. If the spider scrapes more than that amount if items and those items are passed by the item pipeline, the spider will be closed with the reason ``closespider_itempassed``. If zero (or -non set) spiders won't be closed by number of passed items. +non set), spiders won't be closed by number of passed items. StatsMailer extension ~~~~~~~~~~~~~~~~~~~~~ @@ -322,7 +322,7 @@ StatsMailer extension .. class:: scrapy.contrib.statsmailer.StatsMailer -This simple extension can be used to send a notification email every time a +This simple extension can be used to send a notification e-mail every time a domain has finished scraping, including the Scrapy stats collected. The email will be sent to all recipients specified in the :setting:`STATSMAILER_RCPTS` setting. diff --git a/docs/topics/firebug.rst b/docs/topics/firebug.rst index 9f198a410..3e3cd94d0 100644 --- a/docs/topics/firebug.rst +++ b/docs/topics/firebug.rst @@ -38,7 +38,7 @@ In the following screenshot you can see the `Inspect Element`_ tool in action. At first sight, we can see that the directory is divided in categories, which are also divided in subcategories. -However, it seems as if there are more subcategories than the ones being shown +However, it seems that there are more subcategories than the ones being shown in this page, so we'll keep looking: .. image:: _images/firebug2.png @@ -46,7 +46,7 @@ in this page, so we'll keep looking: :height: 629 :alt: Inspecting elements with Firebug -As expected the subcategories contain links to other subcategories, and also +As expected, the subcategories contain links to other subcategories, and also links to actual websites, which is the purpose of the directory. Getting links to follow @@ -98,7 +98,7 @@ This is how the spider would look so far:: Extracting the data =================== -Now we're gonna write the code to extract data from those pages. +Now we're going to write the code to extract data from those pages. With the help of Firebug, we'll take a look at some page containing links to websites (say http://directory.google.com/Top/Arts/Awards/) and find out how we can @@ -154,8 +154,8 @@ Finally, we can write our ``parse_category()`` method:: yield item -make sure you you may find some elements which appear in Firebug but -doesn't belong to the original HTML, such as the typical case of ```` +Be aware that you may find some elements which appear in Firebug but +not in the original HTML, such as the typical case of ```` elements. or tags which Therefer in page HTML diff --git a/docs/topics/firefox.rst b/docs/topics/firefox.rst index f06633857..f0b8eb594 100644 --- a/docs/topics/firefox.rst +++ b/docs/topics/firefox.rst @@ -4,7 +4,7 @@ Using Firefox for scraping ========================== -Here is a list of tips and advices on using Firefox for scraping, along with a +Here is a list of tips and advice on using Firefox for scraping, along with a list of useful Firefox add-ons to ease the scraping process. .. _topics-firefox-livedom: @@ -44,7 +44,7 @@ Firebug useful for scraping. In particular, its `Inspect Element`_ feature comes very handy when you need to construct the XPaths for extracting data because it allows you to view the HTML code of each page element while moving your mouse -over them. +over it. See :ref:`topics-firebug` for a detailed guide on how to use Firebug with Scrapy. @@ -70,8 +70,8 @@ Firecookie ---------- `Firecookie`_ makes it easier to view and manage cookies. You can use this -extension to create a new cookie, delete existing cookies, see list of cookies -for current site, manage cookies permissions and a lot more. +extension to create a new cookie, delete existing cookies, see a list of cookies +for the current site, manage cookies permissions and a lot more. .. _Firebug: http://getfirebug.com .. _Inspect Element: http://www.youtube.com/watch?v=-pT_pDe54aA diff --git a/docs/topics/images.rst b/docs/topics/images.rst index 5cf9601ab..ca4117d47 100644 --- a/docs/topics/images.rst +++ b/docs/topics/images.rst @@ -7,7 +7,7 @@ Downloading Item Images .. currentmodule:: scrapy.contrib.pipeline.images Scrapy provides an :doc:`item pipeline ` for downloading -images attached to a particular item. For example, when you scrape products and +images attached to a particular item, for example, when you scrape products and also want to download their images locally. This pipeline, called the Images Pipeline and implemented in the @@ -44,16 +44,16 @@ this: 3. When the item reaches the :class:`ImagesPipeline`, the URLs in the ``image_urls`` attribute are scheduled for download using the standard Scrapy scheduler and downloader (which means the scheduler and downloader - middlewares are reused), but higher priority to process them before other - pages to scrape. The item remains "locked" at that particular pipeline stage + middlewares are reused), but with a higher priority, processing them before other + pages are scraped. The item remains "locked" at that particular pipeline stage until the images have finish downloading (or fail for some reason). -4. When the images finish downloading (or fail for some reason) the images gets - another field populated with the path of the images downloaded, for example, +4. When the images finish downloading (or fail for some reason) + another field gets populated with their path, for example, ``image_paths``. This attribute is a list of dictionaries containing - information about the image downloaded, such as the downloaded path, and the - original scraped url. This images in the list of the ``image_paths`` field - would retain the same order of the original ``image_urls`` field, which is + information about the images downloaded, such as the downloaded path, and the + original scraped url. The images in the list of the ``image_paths`` field + will retain the same order of the original ``image_urls`` field, which is useful if you decide to use the first image in the list as the primary image. @@ -83,7 +83,7 @@ Here are the methods that you should override in your custom Images Pipeline: :meth:`~item_completed` method, as a list of 2-element tuples. Each tuple will contain ``(success, image_info_or_failure)`` where: - * ``success`` is a boolean which is ``True`` if the image was downloading + * ``success`` is a boolean which is ``True`` if the image was downloaded successfully or ``False`` if it failed for some reason * ``image_info_or_error`` is a dict containing the following keys (if success @@ -131,7 +131,7 @@ Here are the methods that you should override in your custom Images Pipeline: output that will be sent to subsequent item pipeline stages, so you must return (or drop) the item, as you would in any pipeline. - Here is an example of :meth:`~item_completed` method where we + Here is an example of the :meth:`~item_completed` method where we store the downloaded image paths (passed in results) in the ``image_paths`` item field, and we drop the item if it doesn't contain any images:: @@ -246,7 +246,7 @@ images. .. setting:: IMAGES_THUMBS -In order use this feature you must set :setting:`IMAGES_THUMBS` to a dictionary +In order use this feature, you must set :setting:`IMAGES_THUMBS` to a dictionary where the keys are the thumbnail names and the values are their dimensions. For example:: @@ -293,7 +293,7 @@ For example:: IMAGES_MIN_HEIGHT = 110 IMAGES_MIN_WIDTH = 110 -Note: this size constraints only doesn't affect thumbnail generation at all. +Note: these size constraints don't affect thumbnail generation at all. -By default, there are no size constrains, so all images are precessed. +By default, there are no size constraints, so all images are processed.