#154: Language fixes to the documentation

This commit is contained in:
Lucian Ursu 2010-04-18 23:39:54 -03:00
parent dfdac356af
commit 2f8c052484
9 changed files with 122 additions and 104 deletions

View File

@ -4,7 +4,7 @@
Architecture overview
=====================
This document describes the architecture of Scrapy and how their components
This document describes the architecture of Scrapy and how its components
interact.
Overview
@ -41,12 +41,12 @@ Downloader
----------
The Downloader is responsible for fetching web pages and feeding them to the
engine which, in turns, feeds them to the spiders.
engine which, in turn, feeds them to the spiders.
Spiders
-------
Spiders are custom classes written by Scrapy users to parse response and
Spiders are custom classes written by Scrapy users to parse responses and
extract items (aka scraped items) from them or additional URLs (requests) to
follow. Each spider is able to handle a specific domain (or group of domains).
For more information see :ref:`topics-spiders`.
@ -64,7 +64,7 @@ Downloader middlewares
Downloader middlewares are specific hooks that sit between the Engine and the
Downloader and process requests when they pass from the Engine to the
downloader, and responses that pass from Downloader to the Engine. They provide
Downloader, and responses that pass from Downloader to the Engine. They provide
a convenient mechanism for extending Scrapy functionality by plugging custom
code. For more information see :ref:`topics-downloader-middleware`.
@ -80,7 +80,7 @@ functionality by plugging custom code. For more information see
Scheduler middlewares
---------------------
Spider middlewares are specific hooks that sit between the Engine and the
Scheduler middlewares are specific hooks that sit between the Engine and the
Scheduler and process requests when they pass from the Engine to the Scheduler
and vice-versa. They provide a convenient mechanism for extending Scrapy
functionality by plugging custom code.

View File

@ -35,9 +35,9 @@ where you want to insert the middleware. The order does matter because each
middleware performs a different action and your middleware could depend on some
previous (or subsequent) middleware being applied.
If you want to disable a builtin middleware (the ones defined in
If you want to disable a built-in middleware (the ones defined in
:setting:`DOWNLOADER_MIDDLEWARES_BASE` and enabled by default) you must define it
in your project :setting:`DOWNLOADER_MIDDLEWARES` setting and assign `None`
in your project's :setting:`DOWNLOADER_MIDDLEWARES` setting and assign `None`
as its value. For example, if you want to disable the off-site middleware::
DOWNLOADER_MIDDLEWARES = {
@ -67,24 +67,24 @@ single Python class that defines one or more of the following methods:
:class:`~scrapy.http.Response` object, or a :class:`~scrapy.http.Request`
object.
If returns ``None``, Scrapy will continue processing this request, executing all
If it returns ``None``, Scrapy will continue processing this request, executing all
other middlewares until, finally, the appropriate downloader handler is called
the request performed (and its response downloaded).
If returns a :class:`~scrapy.http.Response` object, Scrapy won't bother
If it returns a :class:`~scrapy.http.Response` object, Scrapy won't bother
calling ANY other request or exception middleware, or the appropriate
download function; it'll return that Response. Response middleware is
always called on every response.
always called on every Response.
If returns a :class:`~scrapy.http.Request` object, the returned request will be
re-scheduled (in the Scheduler) to be downloaded in the future. The callback of
If it returns a :class:`~scrapy.http.Request` object, the returned request will be
rescheduled (in the Scheduler) to be downloaded in the future. The callback of
the original request will always be called. If the new request has a callback
it will be called with the response downloaded, and the output of that callback
will then be passed to the original callback. If the new request doesn't have a
callback, the response downloaded will be just passed to the original request
callback.
If returns an :exc:`~scrapy.core.exceptions.IgnoreRequest` exception, the
If it returns an :exc:`~scrapy.core.exceptions.IgnoreRequest` exception, the
entire request will be dropped completely and its callback never called.
:param request: the request being processed
@ -95,14 +95,14 @@ single Python class that defines one or more of the following methods:
.. method:: process_response(request, response, spider)
meth:`process_response` should return a :class:`~scrapy.http.Response`
:meth:`process_response` should return a :class:`~scrapy.http.Response`
object or raise a :exc:`~scrapy.core.exceptions.IgnoreRequest` exception.
If returns a :class:`~scrapy.http.Response` (it could be the same given
response, or a brand-new one) that response will continue to be processed
If it returns a :class:`~scrapy.http.Response` (it could be the same given
response, or a brand-new one), that response will continue to be processed
with the :meth:`process_response` of the next middleware in the pipeline.
If returns an :exc:`~scrapy.core.exceptions.IgnoreRequest` exception, the
If it returns an :exc:`~scrapy.core.exceptions.IgnoreRequest` exception, the
response will be dropped completely and its callback never called.
:param request: the request that originated the response
@ -130,17 +130,17 @@ single Python class that defines one or more of the following methods:
If it returns a :class:`~scrapy.http.Response` object, the response middleware
kicks in, and won't bother calling any other exception middleware.
If it returns a :class:`~scrapy.http.Request` object, returned request is
used to instruct a immediate redirection. Redirection is handled inside
middleware scope, and the original request won't finish until redirected
request is completed. This stop :meth:`process_download_exception`
middleware as returning Response would do.
If it returns a :class:`~scrapy.http.Request` object, the returned request is
used to instruct an immediate redirection.
The original request won't finish until the redirected
request is completed. This stops the :meth:`process_download_exception`
middleware the same as returning Response would do.
:param request: the request that generated the exception
:type request: is a :class:`~scrapy.http.Request` object
:param exception: the raised exception
:type exception: a ``Exception`` object
:type exception: an ``Exception`` object
:param spider: the spider for which this request is intended
:type spider: :class:`~scrapy.spider.BaseSpider` object
@ -190,7 +190,7 @@ HttpAuthMiddleware
This middleware authenticates all requests generated from certain spiders
using `Basic access authentication`_ (aka. HTTP auth).
To enable HTTP authentication from certain spiders set the ``http_user``
To enable HTTP authentication from certain spiders, set the ``http_user``
and ``http_pass`` attributes of those spiders.
Example::
@ -219,7 +219,7 @@ HttpCacheMiddleware
anything from the Internet.
The HTTP cache is useful for testing spiders faster (without having to wait for
downloads every time) and for trying your spider off-line when you don't have
downloads every time) and for trying your spider offline, when you don't have
an Internet connection.
File system storage
@ -227,7 +227,7 @@ File system storage
By default, the :class:`HttpCacheMiddleware` uses a file system storage with the following structure:
Each request/response pair is stored in a different directory containing with
Each request/response pair is stored in a different directory containing
the following files:
* ``request_body`` - the plain request body
@ -235,7 +235,7 @@ the following files:
* ``response_body`` - the plain response body
* ``response_headers`` - the request headers (in raw HTTP format)
* ``meta`` - some metadata of this cache resource in Python ``repr()`` format
(for easy grepeability)
(grep-friendly format)
* ``pickled_meta`` - the same metadata in ``meta`` but pickled for more
efficient deserialization
@ -262,7 +262,7 @@ HTTPCACHE_DIR
Default: ``''`` (empty string)
The directory to use for storing the (low-level) HTTP cache. If empty the HTTP
The directory to use for storing the (low-level) HTTP cache. If empty, the HTTP
cache will be disabled.
.. setting:: HTTPCACHE_EXPIRATION_SECS
@ -274,7 +274,7 @@ Default: ``0``
Number of seconds to use for HTTP cache expiration. Requests that were cached
before this time will be re-downloaded. If zero, cached requests will always
expire. Negative numbers means requests will never expire.
expire. A negative number means requests will never expire.
.. setting:: HTTPCACHE_IGNORE_MISSING
@ -319,7 +319,7 @@ HttpProxyMiddleware
This middleware sets the HTTP proxy to use for requests, by setting the
``proxy`` meta value to :class:`~scrapy.http.Request` objects.
Like the Python standard library modules `urllib`_ and `urllib2`_ it obeys
Like the Python standard library modules `urllib`_ and `urllib2`_, it obeys
the following enviroment variables:
* ``http_proxy``
@ -360,7 +360,7 @@ RetryMiddleware
Failed pages are collected on the scraping process and rescheduled at the
end, once the spider has finished crawling all regular (non failed) pages.
Once there is no more failed pages to retry this middleware sends a signal
Once there are no more failed pages to retry, this middleware sends a signal
(retry_complete), so other extensions could connect to that signal.
The :class:`RetryMiddleware` can be configured through the following
@ -389,7 +389,7 @@ RobotsTxtMiddleware
standard.
To make sure Scrapy respects robots.txt make sure the middleware is enabled
amd the :setting:`ROBOTSTXT_OBEY` setting is enabled.
and the :setting:`ROBOTSTXT_OBEY` setting is enabled.
.. warning:: Keep in mind that, if you crawl using multiple concurrent
requests per domain, Scrapy could still download some forbidden pages
@ -405,7 +405,7 @@ DownloaderStats
.. class:: DownloaderStats
Middleware that store stats of all requests, responses and exceptions that
Middleware that stores stats of all requests, responses and exceptions that
pass through it.
To use this middleware you must enable the :setting:`DOWNLOADER_STATS`

View File

@ -1,25 +1,28 @@
.. _topics-email:
=============
Sending email
Sending e-mail
=============
.. module:: scrapy.mail
:synopsis: Helpers to easily send e-mail.
:synopsis: Email sending facility
Although Python makes sending e-mail relatively easy via the `smtplib`_
library, Scrapy provides its own class for sending emails which is very easy to
use and it's implemented using `Twisted non-blocking IO`_, to avoid affecting
the crawling performance.
Although Python makes sending e-mails relatively easy via the `smtplib`_
library, Scrapy provides its own facility for sending e-mails which is very easy
to use and it's implemented using `Twisted non-blocking IO`_, to avoid
interfering with the non-blocking IO of the crawler.
It's also very easy to configure, having only a few settings.
.. _smtplib: http://docs.python.org/library/smtplib.html
.. _Twisted non-blocking IO: http://twistedmatrix.com/projects/core/documentation/howto/async.html
It also has built-in support for sending attachments.
Quick example
=============
Here's a quick example of how to send an email (without attachments)::
Here's a quick example of how to send an e-mail (without attachments)::
from scrapy.mail import MailSender
@ -34,30 +37,45 @@ uses `Twisted non-blocking IO`_, like the rest of the framework.
.. class:: MailSender(smtphost, mailfrom)
``smtphost`` is a string with the SMTP host to use for sending the emails.
If omitted, :setting:`MAIL_HOST` will be used.
:param smtphost: the SMTP host to use for sending the emails. If omitted, the
:setting:`MAIL_HOST` setting will be used.
:type smtphost: str
``mailfrom`` is a string with the email address to use for sending messages
(in the ``From:`` header). If omitted, :setting:`MAIL_FROM` will be used.
:param mailfrom: the address used to send emails (in the ``From:`` header).
If omitted, the :setting:`MAIL_FROM` setting will be used.
:type mailfrom: str
.. method:: MailSender.send(to, subject, body, cc=None, attachs=())
.. method:: send(to, subject, body, cc=None, attachs=())
Send mail to the given recipients
Send email to the given recipients
``to`` is a list of email recipients
:param to: the e-mail recipients
:type to: list
``subject`` is a string with the subject of the message
:param subject: the subject of the e-mail
:type subject: str
``cc`` is a list of emails to CC
:param cc: the e-mails to CC
:type cc: list
``body`` is a string with the body of the message
:param body: the e-mail body
:type body: str
``attachs`` is an iterable of tuples (attach_name, mimetype, file_object)
where:
``attach_name`` is a string with the name will appear on the emails attachment
``mimetype`` is the mimetype of the attachment
``file_object`` is a readable file object
:param attachs: an iterable of tuples ``(attach_name, mimetype,
file_object)`` where ``attach_name`` is a string with the name that will
appear on the e-mail's attachment, ``mimetype`` is the mimetype of the
attachment and ``file_object`` is a readable file object with the
contents of the attachment
:type attachs: iterable
.. _Twisted non-blocking IO: http://twistedmatrix.com/projects/core/documentation/howto/async.html
MailSender settings
===================
These settings define the default constructor values of the :class:`MailSender`
class, and can be used to configure e-mail notifications in your project without
writing any code (for those extensions that use the :class:`MailSender` class):
* :setting:`MAIL_FROM`
* :setting:`MAIL_HOST`

View File

@ -27,7 +27,7 @@ IgnoreRequest
.. exception:: IgnoreRequest
This exception can be raised by the Scheduler or any downlaoder middleware to
This exception can be raised by the Scheduler or any downloader middleware to
indicate that the request should be ignored.
NotConfigured
@ -36,7 +36,7 @@ NotConfigured
.. exception:: NotConfigured
This exception can be raised by some components to indicate that they will
remain disabled. Those component include:
remain disabled. Those components include:
* Extensions
* Item pipelines

View File

@ -20,10 +20,10 @@ Using Item Exporters
If you are in a hurry, and just want to use an Item Exporter as an :doc:`Item
Pipeline <item-pipeline>` see the :ref:`File Export Pipeline
<file-export-pipeline>`. Otherwise, if you want to know how Item Exporters
work, or need more custom functionality (not covered by the :ref:`File Export
Pipeline <file-export-pipeline>`) continue reading below.
work or need more custom functionality (not covered by the :ref:`File Export
Pipeline <file-export-pipeline>`), continue reading below.
In order to use a Item Exporter, you must instantiate it with its required
In order to use an Item Exporter, you must instantiate it with its required
args. Each Item Exporter requires different arguments, so check each exporter
documentation to be sure, in :ref:`topics-exporters-reference`. After you have
instantiated you exporter, you have to:
@ -72,7 +72,7 @@ Exporter to export scraped items to different files, one per spider::
Serialization of item fields
============================
By default the field values are passed unmodified to the underlying
By default, the field values are passed unmodified to the underlying
serialization library, and the decision of how to serialize them is delegated
to each particular serialization library.
@ -119,7 +119,7 @@ Example::
class ProductXmlExporter(XmlItemExporter):
def serialize_field(self, field, name, value):
if filed == 'price':
if field == 'price':
return '$ %s' % str(value)
return super(Product, self).serialize_field(field, name, value)
@ -241,7 +241,7 @@ XmlItemExporter
</item>
</items>
Unless overriden in :meth:`serialize_field` method, multi-valued fields are
Unless overriden in the :meth:`serialize_field` method, multi-valued fields are
exported by serializing each value inside a ``<value>`` element. This is for
convenience, as multi-valued fields are very common.

View File

@ -4,7 +4,7 @@
Extensions
==========
The extensions framework provide a mechanism for inserting your own
The extensions framework provides a mechanism for inserting your own
custom functionality into Scrapy.
Extensions are just regular classes that are instantiated at Scrapy startup,
@ -50,8 +50,8 @@ orders though, and they are typically irrelevant, ie. it doesn't matter in
which order the extensions are loaded because they don't depend on each other
[1].
However this feature can be exploited if you need to add an extension which
depends on other extension already loaded.
However, this feature can be exploited if you need to add an extension which
depends on other extensions already loaded.
[1] This is is why the :setting:`EXTENSIONS_BASE` setting in Scrapy (which
contains all built-in extensions enabled by default) defines all the extensions
@ -63,7 +63,7 @@ Available, enabled and disabled extensions
Not all available extensions will be enabled. Some of them usually depend on a
particular setting. For example, the HTTP Cache extension is available by default
but disabled unless the :setting:`HTTPCACHE_DIR` setting is set. Both enabled
and disabled extension can be accessed through the
and disabled extensions can be accessed through the
:ref:`topics-extensions-ref-manager`.
Accessing enabled extensions
@ -76,9 +76,9 @@ loaded. For example, to access the ``WebConsole`` extension::
from scrapy.extension import extensions
webconsole_extension = extensions.enabled['WebConsole']
.. seealso::
.. see also::
:ref:`topics-extensions-ref-manager`, for the complete Extension manager
:ref:`topics-extensions-ref-manager`, for the complete Extension Manager
reference.
Writing your own extension
@ -93,7 +93,7 @@ All extension initialization code must be performed in the class constructor
disabled. Otherwise, the extension will be enabled.
Let's take a look at the following example extension which just logs a message
everytime a domain/spider is opened and closed::
every time a domain/spider is opened and closed::
from scrapy.xlib.pydispatch import dispatcher
from scrapy.core import signals
@ -113,7 +113,7 @@ everytime a domain/spider is opened and closed::
.. _topics-extensions-ref-manager:
Extension manager
Extension Manager
=================
.. module:: scrapy.extension
@ -127,7 +127,7 @@ how you :ref:`configure the downloader middlewares
.. class:: ExtensionManager
The extension manager is a singleton object, which is instantiated at module
The Extension Manager is a singleton object, which is instantiated at module
loading time and can be accessed like this::
from scrapy.extension import extensions
@ -190,7 +190,7 @@ Core Stats extension
.. class:: CoreStats
Enable the collection of core statistics, provided the stats collection are
Enable the collection of core statistics, provided the stats collection is
enabled (see :ref:`topics-stats`).
.. _topics-extensions-ref-webconsole:
@ -246,10 +246,10 @@ Memory usage extension
Allows monitoring the memory used by a Scrapy process and:
1, send a notification email when it exceeds a certain value
1, send a notification e-mail when it exceeds a certain value
2. terminate the Scrapy process when it exceeds a certain value
The notification emails can be triggered when a certain warning value is
The notification e-mails can be triggered when a certain warning value is
reached (:setting:`MEMUSAGE_WARNING_MB`) and when the maximum value is reached
(:setting:`MEMUSAGE_LIMIT_MB`) which will also cause the Scrapy process to be
terminated.
@ -271,9 +271,9 @@ Memory debugger extension
.. class:: scrapy.contrib.memdebug.MemoryDebugger
A memory debugger which collects some info about objects uncollected by the
garbage collector and libxml2 memory leaks. To enable this extension turn on
garbage collector and libxml2 memory leaks. To enable this extension, turn on
the :setting:`MEMDEBUG_ENABLED` setting. The report will be printed to standard
output. If the :setting:`MEMDEBUG_NOTIFY` setting contains a list of emails the
output. If the :setting:`MEMDEBUG_NOTIFY` setting contains a list of e-mails the
report will also be sent to those addresses.
Close spider extension
@ -299,7 +299,7 @@ Default: ``0``
An integer which specifies a number of seconds. If the spider remains open for
more than that number of second, it will be automatically closed with the
reason ``closespider_timeout``. If zero (or non set) spiders won't be closed by
reason ``closespider_timeout``. If zero (or non set), spiders won't be closed by
timeout.
.. setting:: CLOSESPIDER_ITEMPASSED
@ -312,7 +312,7 @@ Default: ``0``
An integer which specifies a number of items. If the spider scrapes more than
that amount if items and those items are passed by the item pipeline, the
spider will be closed with the reason ``closespider_itempassed``. If zero (or
non set) spiders won't be closed by number of passed items.
non set), spiders won't be closed by number of passed items.
StatsMailer extension
~~~~~~~~~~~~~~~~~~~~~
@ -322,7 +322,7 @@ StatsMailer extension
.. class:: scrapy.contrib.statsmailer.StatsMailer
This simple extension can be used to send a notification email every time a
This simple extension can be used to send a notification e-mail every time a
domain has finished scraping, including the Scrapy stats collected. The email
will be sent to all recipients specified in the :setting:`STATSMAILER_RCPTS`
setting.

View File

@ -38,7 +38,7 @@ In the following screenshot you can see the `Inspect Element`_ tool in action.
At first sight, we can see that the directory is divided in categories, which
are also divided in subcategories.
However, it seems as if there are more subcategories than the ones being shown
However, it seems that there are more subcategories than the ones being shown
in this page, so we'll keep looking:
.. image:: _images/firebug2.png
@ -46,7 +46,7 @@ in this page, so we'll keep looking:
:height: 629
:alt: Inspecting elements with Firebug
As expected the subcategories contain links to other subcategories, and also
As expected, the subcategories contain links to other subcategories, and also
links to actual websites, which is the purpose of the directory.
Getting links to follow
@ -98,7 +98,7 @@ This is how the spider would look so far::
Extracting the data
===================
Now we're gonna write the code to extract data from those pages.
Now we're going to write the code to extract data from those pages.
With the help of Firebug, we'll take a look at some page containing links to
websites (say http://directory.google.com/Top/Arts/Awards/) and find out how we can
@ -154,8 +154,8 @@ Finally, we can write our ``parse_category()`` method::
yield item
make sure you you may find some elements which appear in Firebug but
doesn't belong to the original HTML, such as the typical case of ``<tbody>``
Be aware that you may find some elements which appear in Firebug but
not in the original HTML, such as the typical case of ``<tbody>``
elements.
or tags which Therefer in page HTML

View File

@ -4,7 +4,7 @@
Using Firefox for scraping
==========================
Here is a list of tips and advices on using Firefox for scraping, along with a
Here is a list of tips and advice on using Firefox for scraping, along with a
list of useful Firefox add-ons to ease the scraping process.
.. _topics-firefox-livedom:
@ -44,7 +44,7 @@ Firebug
useful for scraping. In particular, its `Inspect Element`_ feature comes very
handy when you need to construct the XPaths for extracting data because it
allows you to view the HTML code of each page element while moving your mouse
over them.
over it.
See :ref:`topics-firebug` for a detailed guide on how to use Firebug with
Scrapy.
@ -70,8 +70,8 @@ Firecookie
----------
`Firecookie`_ makes it easier to view and manage cookies. You can use this
extension to create a new cookie, delete existing cookies, see list of cookies
for current site, manage cookies permissions and a lot more.
extension to create a new cookie, delete existing cookies, see a list of cookies
for the current site, manage cookies permissions and a lot more.
.. _Firebug: http://getfirebug.com
.. _Inspect Element: http://www.youtube.com/watch?v=-pT_pDe54aA

View File

@ -7,7 +7,7 @@ Downloading Item Images
.. currentmodule:: scrapy.contrib.pipeline.images
Scrapy provides an :doc:`item pipeline </topics/item-pipeline>` for downloading
images attached to a particular item. For example, when you scrape products and
images attached to a particular item, for example, when you scrape products and
also want to download their images locally.
This pipeline, called the Images Pipeline and implemented in the
@ -44,16 +44,16 @@ this:
3. When the item reaches the :class:`ImagesPipeline`, the URLs in the
``image_urls`` attribute are scheduled for download using the standard
Scrapy scheduler and downloader (which means the scheduler and downloader
middlewares are reused), but higher priority to process them before other
pages to scrape. The item remains "locked" at that particular pipeline stage
middlewares are reused), but with a higher priority, processing them before other
pages are scraped. The item remains "locked" at that particular pipeline stage
until the images have finish downloading (or fail for some reason).
4. When the images finish downloading (or fail for some reason) the images gets
another field populated with the path of the images downloaded, for example,
4. When the images finish downloading (or fail for some reason)
another field gets populated with their path, for example,
``image_paths``. This attribute is a list of dictionaries containing
information about the image downloaded, such as the downloaded path, and the
original scraped url. This images in the list of the ``image_paths`` field
would retain the same order of the original ``image_urls`` field, which is
information about the images downloaded, such as the downloaded path, and the
original scraped url. The images in the list of the ``image_paths`` field
will retain the same order of the original ``image_urls`` field, which is
useful if you decide to use the first image in the list as the primary
image.
@ -83,7 +83,7 @@ Here are the methods that you should override in your custom Images Pipeline:
:meth:`~item_completed` method, as a list of 2-element tuples.
Each tuple will contain ``(success, image_info_or_failure)`` where:
* ``success`` is a boolean which is ``True`` if the image was downloading
* ``success`` is a boolean which is ``True`` if the image was downloaded
successfully or ``False`` if it failed for some reason
* ``image_info_or_error`` is a dict containing the following keys (if success
@ -131,7 +131,7 @@ Here are the methods that you should override in your custom Images Pipeline:
output that will be sent to subsequent item pipeline stages, so you must
return (or drop) the item, as you would in any pipeline.
Here is an example of :meth:`~item_completed` method where we
Here is an example of the :meth:`~item_completed` method where we
store the downloaded image paths (passed in results) in the ``image_paths``
item field, and we drop the item if it doesn't contain any images::
@ -246,7 +246,7 @@ images.
.. setting:: IMAGES_THUMBS
In order use this feature you must set :setting:`IMAGES_THUMBS` to a dictionary
In order use this feature, you must set :setting:`IMAGES_THUMBS` to a dictionary
where the keys are the thumbnail names and the values are their dimensions.
For example::
@ -293,7 +293,7 @@ For example::
IMAGES_MIN_HEIGHT = 110
IMAGES_MIN_WIDTH = 110
Note: this size constraints only doesn't affect thumbnail generation at all.
Note: these size constraints don't affect thumbnail generation at all.
By default, there are no size constrains, so all images are precessed.
By default, there are no size constraints, so all images are processed.