mirror of https://github.com/scrapy/scrapy.git
#154: Language fixes to the documentation
This commit is contained in:
parent
dfdac356af
commit
2f8c052484
|
|
@ -4,7 +4,7 @@
|
|||
Architecture overview
|
||||
=====================
|
||||
|
||||
This document describes the architecture of Scrapy and how their components
|
||||
This document describes the architecture of Scrapy and how its components
|
||||
interact.
|
||||
|
||||
Overview
|
||||
|
|
@ -41,12 +41,12 @@ Downloader
|
|||
----------
|
||||
|
||||
The Downloader is responsible for fetching web pages and feeding them to the
|
||||
engine which, in turns, feeds them to the spiders.
|
||||
engine which, in turn, feeds them to the spiders.
|
||||
|
||||
Spiders
|
||||
-------
|
||||
|
||||
Spiders are custom classes written by Scrapy users to parse response and
|
||||
Spiders are custom classes written by Scrapy users to parse responses and
|
||||
extract items (aka scraped items) from them or additional URLs (requests) to
|
||||
follow. Each spider is able to handle a specific domain (or group of domains).
|
||||
For more information see :ref:`topics-spiders`.
|
||||
|
|
@ -64,7 +64,7 @@ Downloader middlewares
|
|||
|
||||
Downloader middlewares are specific hooks that sit between the Engine and the
|
||||
Downloader and process requests when they pass from the Engine to the
|
||||
downloader, and responses that pass from Downloader to the Engine. They provide
|
||||
Downloader, and responses that pass from Downloader to the Engine. They provide
|
||||
a convenient mechanism for extending Scrapy functionality by plugging custom
|
||||
code. For more information see :ref:`topics-downloader-middleware`.
|
||||
|
||||
|
|
@ -80,7 +80,7 @@ functionality by plugging custom code. For more information see
|
|||
Scheduler middlewares
|
||||
---------------------
|
||||
|
||||
Spider middlewares are specific hooks that sit between the Engine and the
|
||||
Scheduler middlewares are specific hooks that sit between the Engine and the
|
||||
Scheduler and process requests when they pass from the Engine to the Scheduler
|
||||
and vice-versa. They provide a convenient mechanism for extending Scrapy
|
||||
functionality by plugging custom code.
|
||||
|
|
|
|||
|
|
@ -35,9 +35,9 @@ where you want to insert the middleware. The order does matter because each
|
|||
middleware performs a different action and your middleware could depend on some
|
||||
previous (or subsequent) middleware being applied.
|
||||
|
||||
If you want to disable a builtin middleware (the ones defined in
|
||||
If you want to disable a built-in middleware (the ones defined in
|
||||
:setting:`DOWNLOADER_MIDDLEWARES_BASE` and enabled by default) you must define it
|
||||
in your project :setting:`DOWNLOADER_MIDDLEWARES` setting and assign `None`
|
||||
in your project's :setting:`DOWNLOADER_MIDDLEWARES` setting and assign `None`
|
||||
as its value. For example, if you want to disable the off-site middleware::
|
||||
|
||||
DOWNLOADER_MIDDLEWARES = {
|
||||
|
|
@ -67,24 +67,24 @@ single Python class that defines one or more of the following methods:
|
|||
:class:`~scrapy.http.Response` object, or a :class:`~scrapy.http.Request`
|
||||
object.
|
||||
|
||||
If returns ``None``, Scrapy will continue processing this request, executing all
|
||||
If it returns ``None``, Scrapy will continue processing this request, executing all
|
||||
other middlewares until, finally, the appropriate downloader handler is called
|
||||
the request performed (and its response downloaded).
|
||||
|
||||
If returns a :class:`~scrapy.http.Response` object, Scrapy won't bother
|
||||
If it returns a :class:`~scrapy.http.Response` object, Scrapy won't bother
|
||||
calling ANY other request or exception middleware, or the appropriate
|
||||
download function; it'll return that Response. Response middleware is
|
||||
always called on every response.
|
||||
always called on every Response.
|
||||
|
||||
If returns a :class:`~scrapy.http.Request` object, the returned request will be
|
||||
re-scheduled (in the Scheduler) to be downloaded in the future. The callback of
|
||||
If it returns a :class:`~scrapy.http.Request` object, the returned request will be
|
||||
rescheduled (in the Scheduler) to be downloaded in the future. The callback of
|
||||
the original request will always be called. If the new request has a callback
|
||||
it will be called with the response downloaded, and the output of that callback
|
||||
will then be passed to the original callback. If the new request doesn't have a
|
||||
callback, the response downloaded will be just passed to the original request
|
||||
callback.
|
||||
|
||||
If returns an :exc:`~scrapy.core.exceptions.IgnoreRequest` exception, the
|
||||
If it returns an :exc:`~scrapy.core.exceptions.IgnoreRequest` exception, the
|
||||
entire request will be dropped completely and its callback never called.
|
||||
|
||||
:param request: the request being processed
|
||||
|
|
@ -95,14 +95,14 @@ single Python class that defines one or more of the following methods:
|
|||
|
||||
.. method:: process_response(request, response, spider)
|
||||
|
||||
meth:`process_response` should return a :class:`~scrapy.http.Response`
|
||||
:meth:`process_response` should return a :class:`~scrapy.http.Response`
|
||||
object or raise a :exc:`~scrapy.core.exceptions.IgnoreRequest` exception.
|
||||
|
||||
If returns a :class:`~scrapy.http.Response` (it could be the same given
|
||||
response, or a brand-new one) that response will continue to be processed
|
||||
If it returns a :class:`~scrapy.http.Response` (it could be the same given
|
||||
response, or a brand-new one), that response will continue to be processed
|
||||
with the :meth:`process_response` of the next middleware in the pipeline.
|
||||
|
||||
If returns an :exc:`~scrapy.core.exceptions.IgnoreRequest` exception, the
|
||||
If it returns an :exc:`~scrapy.core.exceptions.IgnoreRequest` exception, the
|
||||
response will be dropped completely and its callback never called.
|
||||
|
||||
:param request: the request that originated the response
|
||||
|
|
@ -130,17 +130,17 @@ single Python class that defines one or more of the following methods:
|
|||
If it returns a :class:`~scrapy.http.Response` object, the response middleware
|
||||
kicks in, and won't bother calling any other exception middleware.
|
||||
|
||||
If it returns a :class:`~scrapy.http.Request` object, returned request is
|
||||
used to instruct a immediate redirection. Redirection is handled inside
|
||||
middleware scope, and the original request won't finish until redirected
|
||||
request is completed. This stop :meth:`process_download_exception`
|
||||
middleware as returning Response would do.
|
||||
If it returns a :class:`~scrapy.http.Request` object, the returned request is
|
||||
used to instruct an immediate redirection.
|
||||
The original request won't finish until the redirected
|
||||
request is completed. This stops the :meth:`process_download_exception`
|
||||
middleware the same as returning Response would do.
|
||||
|
||||
:param request: the request that generated the exception
|
||||
:type request: is a :class:`~scrapy.http.Request` object
|
||||
|
||||
:param exception: the raised exception
|
||||
:type exception: a ``Exception`` object
|
||||
:type exception: an ``Exception`` object
|
||||
|
||||
:param spider: the spider for which this request is intended
|
||||
:type spider: :class:`~scrapy.spider.BaseSpider` object
|
||||
|
|
@ -190,7 +190,7 @@ HttpAuthMiddleware
|
|||
This middleware authenticates all requests generated from certain spiders
|
||||
using `Basic access authentication`_ (aka. HTTP auth).
|
||||
|
||||
To enable HTTP authentication from certain spiders set the ``http_user``
|
||||
To enable HTTP authentication from certain spiders, set the ``http_user``
|
||||
and ``http_pass`` attributes of those spiders.
|
||||
|
||||
Example::
|
||||
|
|
@ -219,7 +219,7 @@ HttpCacheMiddleware
|
|||
anything from the Internet.
|
||||
|
||||
The HTTP cache is useful for testing spiders faster (without having to wait for
|
||||
downloads every time) and for trying your spider off-line when you don't have
|
||||
downloads every time) and for trying your spider offline, when you don't have
|
||||
an Internet connection.
|
||||
|
||||
File system storage
|
||||
|
|
@ -227,7 +227,7 @@ File system storage
|
|||
|
||||
By default, the :class:`HttpCacheMiddleware` uses a file system storage with the following structure:
|
||||
|
||||
Each request/response pair is stored in a different directory containing with
|
||||
Each request/response pair is stored in a different directory containing
|
||||
the following files:
|
||||
|
||||
* ``request_body`` - the plain request body
|
||||
|
|
@ -235,7 +235,7 @@ the following files:
|
|||
* ``response_body`` - the plain response body
|
||||
* ``response_headers`` - the request headers (in raw HTTP format)
|
||||
* ``meta`` - some metadata of this cache resource in Python ``repr()`` format
|
||||
(for easy grepeability)
|
||||
(grep-friendly format)
|
||||
* ``pickled_meta`` - the same metadata in ``meta`` but pickled for more
|
||||
efficient deserialization
|
||||
|
||||
|
|
@ -262,7 +262,7 @@ HTTPCACHE_DIR
|
|||
|
||||
Default: ``''`` (empty string)
|
||||
|
||||
The directory to use for storing the (low-level) HTTP cache. If empty the HTTP
|
||||
The directory to use for storing the (low-level) HTTP cache. If empty, the HTTP
|
||||
cache will be disabled.
|
||||
|
||||
.. setting:: HTTPCACHE_EXPIRATION_SECS
|
||||
|
|
@ -274,7 +274,7 @@ Default: ``0``
|
|||
|
||||
Number of seconds to use for HTTP cache expiration. Requests that were cached
|
||||
before this time will be re-downloaded. If zero, cached requests will always
|
||||
expire. Negative numbers means requests will never expire.
|
||||
expire. A negative number means requests will never expire.
|
||||
|
||||
.. setting:: HTTPCACHE_IGNORE_MISSING
|
||||
|
||||
|
|
@ -319,7 +319,7 @@ HttpProxyMiddleware
|
|||
This middleware sets the HTTP proxy to use for requests, by setting the
|
||||
``proxy`` meta value to :class:`~scrapy.http.Request` objects.
|
||||
|
||||
Like the Python standard library modules `urllib`_ and `urllib2`_ it obeys
|
||||
Like the Python standard library modules `urllib`_ and `urllib2`_, it obeys
|
||||
the following enviroment variables:
|
||||
|
||||
* ``http_proxy``
|
||||
|
|
@ -360,7 +360,7 @@ RetryMiddleware
|
|||
|
||||
Failed pages are collected on the scraping process and rescheduled at the
|
||||
end, once the spider has finished crawling all regular (non failed) pages.
|
||||
Once there is no more failed pages to retry this middleware sends a signal
|
||||
Once there are no more failed pages to retry, this middleware sends a signal
|
||||
(retry_complete), so other extensions could connect to that signal.
|
||||
|
||||
The :class:`RetryMiddleware` can be configured through the following
|
||||
|
|
@ -389,7 +389,7 @@ RobotsTxtMiddleware
|
|||
standard.
|
||||
|
||||
To make sure Scrapy respects robots.txt make sure the middleware is enabled
|
||||
amd the :setting:`ROBOTSTXT_OBEY` setting is enabled.
|
||||
and the :setting:`ROBOTSTXT_OBEY` setting is enabled.
|
||||
|
||||
.. warning:: Keep in mind that, if you crawl using multiple concurrent
|
||||
requests per domain, Scrapy could still download some forbidden pages
|
||||
|
|
@ -405,7 +405,7 @@ DownloaderStats
|
|||
|
||||
.. class:: DownloaderStats
|
||||
|
||||
Middleware that store stats of all requests, responses and exceptions that
|
||||
Middleware that stores stats of all requests, responses and exceptions that
|
||||
pass through it.
|
||||
|
||||
To use this middleware you must enable the :setting:`DOWNLOADER_STATS`
|
||||
|
|
|
|||
|
|
@ -1,25 +1,28 @@
|
|||
.. _topics-email:
|
||||
|
||||
=============
|
||||
Sending email
|
||||
Sending e-mail
|
||||
=============
|
||||
|
||||
.. module:: scrapy.mail
|
||||
:synopsis: Helpers to easily send e-mail.
|
||||
:synopsis: Email sending facility
|
||||
|
||||
Although Python makes sending e-mail relatively easy via the `smtplib`_
|
||||
library, Scrapy provides its own class for sending emails which is very easy to
|
||||
use and it's implemented using `Twisted non-blocking IO`_, to avoid affecting
|
||||
the crawling performance.
|
||||
Although Python makes sending e-mails relatively easy via the `smtplib`_
|
||||
library, Scrapy provides its own facility for sending e-mails which is very easy
|
||||
to use and it's implemented using `Twisted non-blocking IO`_, to avoid
|
||||
interfering with the non-blocking IO of the crawler.
|
||||
|
||||
It's also very easy to configure, having only a few settings.
|
||||
|
||||
.. _smtplib: http://docs.python.org/library/smtplib.html
|
||||
.. _Twisted non-blocking IO: http://twistedmatrix.com/projects/core/documentation/howto/async.html
|
||||
|
||||
It also has built-in support for sending attachments.
|
||||
|
||||
Quick example
|
||||
=============
|
||||
|
||||
Here's a quick example of how to send an email (without attachments)::
|
||||
Here's a quick example of how to send an e-mail (without attachments)::
|
||||
|
||||
from scrapy.mail import MailSender
|
||||
|
||||
|
|
@ -34,30 +37,45 @@ uses `Twisted non-blocking IO`_, like the rest of the framework.
|
|||
|
||||
.. class:: MailSender(smtphost, mailfrom)
|
||||
|
||||
``smtphost`` is a string with the SMTP host to use for sending the emails.
|
||||
If omitted, :setting:`MAIL_HOST` will be used.
|
||||
:param smtphost: the SMTP host to use for sending the emails. If omitted, the
|
||||
:setting:`MAIL_HOST` setting will be used.
|
||||
:type smtphost: str
|
||||
|
||||
``mailfrom`` is a string with the email address to use for sending messages
|
||||
(in the ``From:`` header). If omitted, :setting:`MAIL_FROM` will be used.
|
||||
:param mailfrom: the address used to send emails (in the ``From:`` header).
|
||||
If omitted, the :setting:`MAIL_FROM` setting will be used.
|
||||
:type mailfrom: str
|
||||
|
||||
.. method:: MailSender.send(to, subject, body, cc=None, attachs=())
|
||||
.. method:: send(to, subject, body, cc=None, attachs=())
|
||||
|
||||
Send mail to the given recipients
|
||||
Send email to the given recipients
|
||||
|
||||
``to`` is a list of email recipients
|
||||
:param to: the e-mail recipients
|
||||
:type to: list
|
||||
|
||||
``subject`` is a string with the subject of the message
|
||||
:param subject: the subject of the e-mail
|
||||
:type subject: str
|
||||
|
||||
``cc`` is a list of emails to CC
|
||||
:param cc: the e-mails to CC
|
||||
:type cc: list
|
||||
|
||||
``body`` is a string with the body of the message
|
||||
:param body: the e-mail body
|
||||
:type body: str
|
||||
|
||||
``attachs`` is an iterable of tuples (attach_name, mimetype, file_object)
|
||||
where:
|
||||
|
||||
``attach_name`` is a string with the name will appear on the emails attachment
|
||||
``mimetype`` is the mimetype of the attachment
|
||||
``file_object`` is a readable file object
|
||||
:param attachs: an iterable of tuples ``(attach_name, mimetype,
|
||||
file_object)`` where ``attach_name`` is a string with the name that will
|
||||
appear on the e-mail's attachment, ``mimetype`` is the mimetype of the
|
||||
attachment and ``file_object`` is a readable file object with the
|
||||
contents of the attachment
|
||||
:type attachs: iterable
|
||||
|
||||
|
||||
.. _Twisted non-blocking IO: http://twistedmatrix.com/projects/core/documentation/howto/async.html
|
||||
MailSender settings
|
||||
===================
|
||||
|
||||
These settings define the default constructor values of the :class:`MailSender`
|
||||
class, and can be used to configure e-mail notifications in your project without
|
||||
writing any code (for those extensions that use the :class:`MailSender` class):
|
||||
|
||||
* :setting:`MAIL_FROM`
|
||||
* :setting:`MAIL_HOST`
|
||||
|
||||
|
|
|
|||
|
|
@ -27,7 +27,7 @@ IgnoreRequest
|
|||
|
||||
.. exception:: IgnoreRequest
|
||||
|
||||
This exception can be raised by the Scheduler or any downlaoder middleware to
|
||||
This exception can be raised by the Scheduler or any downloader middleware to
|
||||
indicate that the request should be ignored.
|
||||
|
||||
NotConfigured
|
||||
|
|
@ -36,7 +36,7 @@ NotConfigured
|
|||
.. exception:: NotConfigured
|
||||
|
||||
This exception can be raised by some components to indicate that they will
|
||||
remain disabled. Those component include:
|
||||
remain disabled. Those components include:
|
||||
|
||||
* Extensions
|
||||
* Item pipelines
|
||||
|
|
|
|||
|
|
@ -20,10 +20,10 @@ Using Item Exporters
|
|||
If you are in a hurry, and just want to use an Item Exporter as an :doc:`Item
|
||||
Pipeline <item-pipeline>` see the :ref:`File Export Pipeline
|
||||
<file-export-pipeline>`. Otherwise, if you want to know how Item Exporters
|
||||
work, or need more custom functionality (not covered by the :ref:`File Export
|
||||
Pipeline <file-export-pipeline>`) continue reading below.
|
||||
work or need more custom functionality (not covered by the :ref:`File Export
|
||||
Pipeline <file-export-pipeline>`), continue reading below.
|
||||
|
||||
In order to use a Item Exporter, you must instantiate it with its required
|
||||
In order to use an Item Exporter, you must instantiate it with its required
|
||||
args. Each Item Exporter requires different arguments, so check each exporter
|
||||
documentation to be sure, in :ref:`topics-exporters-reference`. After you have
|
||||
instantiated you exporter, you have to:
|
||||
|
|
@ -72,7 +72,7 @@ Exporter to export scraped items to different files, one per spider::
|
|||
Serialization of item fields
|
||||
============================
|
||||
|
||||
By default the field values are passed unmodified to the underlying
|
||||
By default, the field values are passed unmodified to the underlying
|
||||
serialization library, and the decision of how to serialize them is delegated
|
||||
to each particular serialization library.
|
||||
|
||||
|
|
@ -119,7 +119,7 @@ Example::
|
|||
class ProductXmlExporter(XmlItemExporter):
|
||||
|
||||
def serialize_field(self, field, name, value):
|
||||
if filed == 'price':
|
||||
if field == 'price':
|
||||
return '$ %s' % str(value)
|
||||
return super(Product, self).serialize_field(field, name, value)
|
||||
|
||||
|
|
@ -241,7 +241,7 @@ XmlItemExporter
|
|||
</item>
|
||||
</items>
|
||||
|
||||
Unless overriden in :meth:`serialize_field` method, multi-valued fields are
|
||||
Unless overriden in the :meth:`serialize_field` method, multi-valued fields are
|
||||
exported by serializing each value inside a ``<value>`` element. This is for
|
||||
convenience, as multi-valued fields are very common.
|
||||
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@
|
|||
Extensions
|
||||
==========
|
||||
|
||||
The extensions framework provide a mechanism for inserting your own
|
||||
The extensions framework provides a mechanism for inserting your own
|
||||
custom functionality into Scrapy.
|
||||
|
||||
Extensions are just regular classes that are instantiated at Scrapy startup,
|
||||
|
|
@ -50,8 +50,8 @@ orders though, and they are typically irrelevant, ie. it doesn't matter in
|
|||
which order the extensions are loaded because they don't depend on each other
|
||||
[1].
|
||||
|
||||
However this feature can be exploited if you need to add an extension which
|
||||
depends on other extension already loaded.
|
||||
However, this feature can be exploited if you need to add an extension which
|
||||
depends on other extensions already loaded.
|
||||
|
||||
[1] This is is why the :setting:`EXTENSIONS_BASE` setting in Scrapy (which
|
||||
contains all built-in extensions enabled by default) defines all the extensions
|
||||
|
|
@ -63,7 +63,7 @@ Available, enabled and disabled extensions
|
|||
Not all available extensions will be enabled. Some of them usually depend on a
|
||||
particular setting. For example, the HTTP Cache extension is available by default
|
||||
but disabled unless the :setting:`HTTPCACHE_DIR` setting is set. Both enabled
|
||||
and disabled extension can be accessed through the
|
||||
and disabled extensions can be accessed through the
|
||||
:ref:`topics-extensions-ref-manager`.
|
||||
|
||||
Accessing enabled extensions
|
||||
|
|
@ -76,9 +76,9 @@ loaded. For example, to access the ``WebConsole`` extension::
|
|||
from scrapy.extension import extensions
|
||||
webconsole_extension = extensions.enabled['WebConsole']
|
||||
|
||||
.. seealso::
|
||||
.. see also::
|
||||
|
||||
:ref:`topics-extensions-ref-manager`, for the complete Extension manager
|
||||
:ref:`topics-extensions-ref-manager`, for the complete Extension Manager
|
||||
reference.
|
||||
|
||||
Writing your own extension
|
||||
|
|
@ -93,7 +93,7 @@ All extension initialization code must be performed in the class constructor
|
|||
disabled. Otherwise, the extension will be enabled.
|
||||
|
||||
Let's take a look at the following example extension which just logs a message
|
||||
everytime a domain/spider is opened and closed::
|
||||
every time a domain/spider is opened and closed::
|
||||
|
||||
from scrapy.xlib.pydispatch import dispatcher
|
||||
from scrapy.core import signals
|
||||
|
|
@ -113,7 +113,7 @@ everytime a domain/spider is opened and closed::
|
|||
|
||||
.. _topics-extensions-ref-manager:
|
||||
|
||||
Extension manager
|
||||
Extension Manager
|
||||
=================
|
||||
|
||||
.. module:: scrapy.extension
|
||||
|
|
@ -127,7 +127,7 @@ how you :ref:`configure the downloader middlewares
|
|||
|
||||
.. class:: ExtensionManager
|
||||
|
||||
The extension manager is a singleton object, which is instantiated at module
|
||||
The Extension Manager is a singleton object, which is instantiated at module
|
||||
loading time and can be accessed like this::
|
||||
|
||||
from scrapy.extension import extensions
|
||||
|
|
@ -190,7 +190,7 @@ Core Stats extension
|
|||
|
||||
.. class:: CoreStats
|
||||
|
||||
Enable the collection of core statistics, provided the stats collection are
|
||||
Enable the collection of core statistics, provided the stats collection is
|
||||
enabled (see :ref:`topics-stats`).
|
||||
|
||||
.. _topics-extensions-ref-webconsole:
|
||||
|
|
@ -246,10 +246,10 @@ Memory usage extension
|
|||
|
||||
Allows monitoring the memory used by a Scrapy process and:
|
||||
|
||||
1, send a notification email when it exceeds a certain value
|
||||
1, send a notification e-mail when it exceeds a certain value
|
||||
2. terminate the Scrapy process when it exceeds a certain value
|
||||
|
||||
The notification emails can be triggered when a certain warning value is
|
||||
The notification e-mails can be triggered when a certain warning value is
|
||||
reached (:setting:`MEMUSAGE_WARNING_MB`) and when the maximum value is reached
|
||||
(:setting:`MEMUSAGE_LIMIT_MB`) which will also cause the Scrapy process to be
|
||||
terminated.
|
||||
|
|
@ -271,9 +271,9 @@ Memory debugger extension
|
|||
.. class:: scrapy.contrib.memdebug.MemoryDebugger
|
||||
|
||||
A memory debugger which collects some info about objects uncollected by the
|
||||
garbage collector and libxml2 memory leaks. To enable this extension turn on
|
||||
garbage collector and libxml2 memory leaks. To enable this extension, turn on
|
||||
the :setting:`MEMDEBUG_ENABLED` setting. The report will be printed to standard
|
||||
output. If the :setting:`MEMDEBUG_NOTIFY` setting contains a list of emails the
|
||||
output. If the :setting:`MEMDEBUG_NOTIFY` setting contains a list of e-mails the
|
||||
report will also be sent to those addresses.
|
||||
|
||||
Close spider extension
|
||||
|
|
@ -299,7 +299,7 @@ Default: ``0``
|
|||
|
||||
An integer which specifies a number of seconds. If the spider remains open for
|
||||
more than that number of second, it will be automatically closed with the
|
||||
reason ``closespider_timeout``. If zero (or non set) spiders won't be closed by
|
||||
reason ``closespider_timeout``. If zero (or non set), spiders won't be closed by
|
||||
timeout.
|
||||
|
||||
.. setting:: CLOSESPIDER_ITEMPASSED
|
||||
|
|
@ -312,7 +312,7 @@ Default: ``0``
|
|||
An integer which specifies a number of items. If the spider scrapes more than
|
||||
that amount if items and those items are passed by the item pipeline, the
|
||||
spider will be closed with the reason ``closespider_itempassed``. If zero (or
|
||||
non set) spiders won't be closed by number of passed items.
|
||||
non set), spiders won't be closed by number of passed items.
|
||||
|
||||
StatsMailer extension
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
|
|
@ -322,7 +322,7 @@ StatsMailer extension
|
|||
|
||||
.. class:: scrapy.contrib.statsmailer.StatsMailer
|
||||
|
||||
This simple extension can be used to send a notification email every time a
|
||||
This simple extension can be used to send a notification e-mail every time a
|
||||
domain has finished scraping, including the Scrapy stats collected. The email
|
||||
will be sent to all recipients specified in the :setting:`STATSMAILER_RCPTS`
|
||||
setting.
|
||||
|
|
|
|||
|
|
@ -38,7 +38,7 @@ In the following screenshot you can see the `Inspect Element`_ tool in action.
|
|||
At first sight, we can see that the directory is divided in categories, which
|
||||
are also divided in subcategories.
|
||||
|
||||
However, it seems as if there are more subcategories than the ones being shown
|
||||
However, it seems that there are more subcategories than the ones being shown
|
||||
in this page, so we'll keep looking:
|
||||
|
||||
.. image:: _images/firebug2.png
|
||||
|
|
@ -46,7 +46,7 @@ in this page, so we'll keep looking:
|
|||
:height: 629
|
||||
:alt: Inspecting elements with Firebug
|
||||
|
||||
As expected the subcategories contain links to other subcategories, and also
|
||||
As expected, the subcategories contain links to other subcategories, and also
|
||||
links to actual websites, which is the purpose of the directory.
|
||||
|
||||
Getting links to follow
|
||||
|
|
@ -98,7 +98,7 @@ This is how the spider would look so far::
|
|||
Extracting the data
|
||||
===================
|
||||
|
||||
Now we're gonna write the code to extract data from those pages.
|
||||
Now we're going to write the code to extract data from those pages.
|
||||
|
||||
With the help of Firebug, we'll take a look at some page containing links to
|
||||
websites (say http://directory.google.com/Top/Arts/Awards/) and find out how we can
|
||||
|
|
@ -154,8 +154,8 @@ Finally, we can write our ``parse_category()`` method::
|
|||
yield item
|
||||
|
||||
|
||||
make sure you you may find some elements which appear in Firebug but
|
||||
doesn't belong to the original HTML, such as the typical case of ``<tbody>``
|
||||
Be aware that you may find some elements which appear in Firebug but
|
||||
not in the original HTML, such as the typical case of ``<tbody>``
|
||||
elements.
|
||||
|
||||
or tags which Therefer in page HTML
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@
|
|||
Using Firefox for scraping
|
||||
==========================
|
||||
|
||||
Here is a list of tips and advices on using Firefox for scraping, along with a
|
||||
Here is a list of tips and advice on using Firefox for scraping, along with a
|
||||
list of useful Firefox add-ons to ease the scraping process.
|
||||
|
||||
.. _topics-firefox-livedom:
|
||||
|
|
@ -44,7 +44,7 @@ Firebug
|
|||
useful for scraping. In particular, its `Inspect Element`_ feature comes very
|
||||
handy when you need to construct the XPaths for extracting data because it
|
||||
allows you to view the HTML code of each page element while moving your mouse
|
||||
over them.
|
||||
over it.
|
||||
|
||||
See :ref:`topics-firebug` for a detailed guide on how to use Firebug with
|
||||
Scrapy.
|
||||
|
|
@ -70,8 +70,8 @@ Firecookie
|
|||
----------
|
||||
|
||||
`Firecookie`_ makes it easier to view and manage cookies. You can use this
|
||||
extension to create a new cookie, delete existing cookies, see list of cookies
|
||||
for current site, manage cookies permissions and a lot more.
|
||||
extension to create a new cookie, delete existing cookies, see a list of cookies
|
||||
for the current site, manage cookies permissions and a lot more.
|
||||
|
||||
.. _Firebug: http://getfirebug.com
|
||||
.. _Inspect Element: http://www.youtube.com/watch?v=-pT_pDe54aA
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@ Downloading Item Images
|
|||
.. currentmodule:: scrapy.contrib.pipeline.images
|
||||
|
||||
Scrapy provides an :doc:`item pipeline </topics/item-pipeline>` for downloading
|
||||
images attached to a particular item. For example, when you scrape products and
|
||||
images attached to a particular item, for example, when you scrape products and
|
||||
also want to download their images locally.
|
||||
|
||||
This pipeline, called the Images Pipeline and implemented in the
|
||||
|
|
@ -44,16 +44,16 @@ this:
|
|||
3. When the item reaches the :class:`ImagesPipeline`, the URLs in the
|
||||
``image_urls`` attribute are scheduled for download using the standard
|
||||
Scrapy scheduler and downloader (which means the scheduler and downloader
|
||||
middlewares are reused), but higher priority to process them before other
|
||||
pages to scrape. The item remains "locked" at that particular pipeline stage
|
||||
middlewares are reused), but with a higher priority, processing them before other
|
||||
pages are scraped. The item remains "locked" at that particular pipeline stage
|
||||
until the images have finish downloading (or fail for some reason).
|
||||
|
||||
4. When the images finish downloading (or fail for some reason) the images gets
|
||||
another field populated with the path of the images downloaded, for example,
|
||||
4. When the images finish downloading (or fail for some reason)
|
||||
another field gets populated with their path, for example,
|
||||
``image_paths``. This attribute is a list of dictionaries containing
|
||||
information about the image downloaded, such as the downloaded path, and the
|
||||
original scraped url. This images in the list of the ``image_paths`` field
|
||||
would retain the same order of the original ``image_urls`` field, which is
|
||||
information about the images downloaded, such as the downloaded path, and the
|
||||
original scraped url. The images in the list of the ``image_paths`` field
|
||||
will retain the same order of the original ``image_urls`` field, which is
|
||||
useful if you decide to use the first image in the list as the primary
|
||||
image.
|
||||
|
||||
|
|
@ -83,7 +83,7 @@ Here are the methods that you should override in your custom Images Pipeline:
|
|||
:meth:`~item_completed` method, as a list of 2-element tuples.
|
||||
Each tuple will contain ``(success, image_info_or_failure)`` where:
|
||||
|
||||
* ``success`` is a boolean which is ``True`` if the image was downloading
|
||||
* ``success`` is a boolean which is ``True`` if the image was downloaded
|
||||
successfully or ``False`` if it failed for some reason
|
||||
|
||||
* ``image_info_or_error`` is a dict containing the following keys (if success
|
||||
|
|
@ -131,7 +131,7 @@ Here are the methods that you should override in your custom Images Pipeline:
|
|||
output that will be sent to subsequent item pipeline stages, so you must
|
||||
return (or drop) the item, as you would in any pipeline.
|
||||
|
||||
Here is an example of :meth:`~item_completed` method where we
|
||||
Here is an example of the :meth:`~item_completed` method where we
|
||||
store the downloaded image paths (passed in results) in the ``image_paths``
|
||||
item field, and we drop the item if it doesn't contain any images::
|
||||
|
||||
|
|
@ -246,7 +246,7 @@ images.
|
|||
|
||||
.. setting:: IMAGES_THUMBS
|
||||
|
||||
In order use this feature you must set :setting:`IMAGES_THUMBS` to a dictionary
|
||||
In order use this feature, you must set :setting:`IMAGES_THUMBS` to a dictionary
|
||||
where the keys are the thumbnail names and the values are their dimensions.
|
||||
|
||||
For example::
|
||||
|
|
@ -293,7 +293,7 @@ For example::
|
|||
IMAGES_MIN_HEIGHT = 110
|
||||
IMAGES_MIN_WIDTH = 110
|
||||
|
||||
Note: this size constraints only doesn't affect thumbnail generation at all.
|
||||
Note: these size constraints don't affect thumbnail generation at all.
|
||||
|
||||
By default, there are no size constrains, so all images are precessed.
|
||||
By default, there are no size constraints, so all images are processed.
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue