finished cleaning up closedomain documentation, and updated default settings

This commit is contained in:
Pablo Hoffman 2009-07-27 15:42:35 -03:00
parent 09ba6927d7
commit fcc91901eb
3 changed files with 25 additions and 33 deletions

View File

@ -139,18 +139,32 @@ Close domain extension
Closes a domain/spider automatically when some conditions are met, using a
specific closing reason for each condition.
The supported conditions for closing a domain can be configured thorough these
settings:
The conditions for closing a domain can be configured through the following
settings. Other conditions will be supported in the future.
* :setting:`CLOSEDOMAIN_TIMEOUT` - an integer which specifies a number of
seconds. If the domain remains open for more than that time, it will be
closed with the reason ``closedomain_timeout``.
* :setting:`CLOSEDOMAIN_ITEMPASSED` - an integer which specifies a number of
items. If the spider scrapes more than that amount if items and those items
are passed by the item pipeline, the domain will be closed with the reason
``closedomain_itempassed``.
.. setting:: CLOSEDOMAIN_TIMEOUT
Other conditions will be supported in the future.
CLOSEDOMAIN_TIMEOUT
~~~~~~~~~~~~~~~~~~~
Default: ``0``
An integer which specifies a number of seconds. If the domain remains open for
more than that number of second, it will be automatically closed with the
reason ``closedomain_timeout``. If zero (or non set) domains won't be closed by
timeout.
.. setting:: CLOSEDOMAIN_ITEMPASSED
CLOSEDOMAIN_ITEMPASSED
~~~~~~~~~~~~~~~~~~~~~~
Default: ``0``
An integer which specifies a number of items. If the spider scrapes more than
that amount if items and those items are passed by the item pipeline, the
domain will be closed with the reason ``closedomain_itempassed``. If zero (or
non set) domains won't be closed by number of passed items.
Stack trace dump extension
---------------------------

View File

@ -81,28 +81,6 @@ Default: ``True``
Whether to split HTTP cache storage in several dirs for performance.
.. setting:: CLOSEDOMAIN_NOTIFY
CLOSEDOMAIN_NOTIFY
------------------
Default: ``[]``
Scope: ``scrapy.contrib.closedomain``
A list of emails to notify if the domain has been automatically closed by timeout.
.. setting:: CLOSEDOMAIN_TIMEOUT
CLOSEDOMAIN_TIMEOUT
-------------------
Default: ``0``
Scope: ``scrapy.contrib.closedomain``
A timeout (in secs) for automatically closing a spider. Spiders that remain
open for more than this time will be automatically closed. If zero, the
automatically closing is disabled.
.. setting:: CLUSTER_LOGDIR
CLUSTER_LOGDIR

View File

@ -21,7 +21,7 @@ BOT_NAME = 'scrapybot'
BOT_VERSION = '1.0'
CLOSEDOMAIN_TIMEOUT = 0
CLOSEDOMAIN_NOTIFY = []
CLOSEDOMAIN_ITEMPASSED = 0
CLUSTER_LOGDIR = ''