diff --git a/docs/ref/extensions.rst b/docs/ref/extensions.rst index a32034561..3402fcf35 100644 --- a/docs/ref/extensions.rst +++ b/docs/ref/extensions.rst @@ -139,18 +139,32 @@ Close domain extension Closes a domain/spider automatically when some conditions are met, using a specific closing reason for each condition. -The supported conditions for closing a domain can be configured thorough these -settings: +The conditions for closing a domain can be configured through the following +settings. Other conditions will be supported in the future. -* :setting:`CLOSEDOMAIN_TIMEOUT` - an integer which specifies a number of - seconds. If the domain remains open for more than that time, it will be - closed with the reason ``closedomain_timeout``. -* :setting:`CLOSEDOMAIN_ITEMPASSED` - an integer which specifies a number of - items. If the spider scrapes more than that amount if items and those items - are passed by the item pipeline, the domain will be closed with the reason - ``closedomain_itempassed``. +.. setting:: CLOSEDOMAIN_TIMEOUT -Other conditions will be supported in the future. +CLOSEDOMAIN_TIMEOUT +~~~~~~~~~~~~~~~~~~~ + +Default: ``0`` + +An integer which specifies a number of seconds. If the domain remains open for +more than that number of second, it will be automatically closed with the +reason ``closedomain_timeout``. If zero (or non set) domains won't be closed by +timeout. + +.. setting:: CLOSEDOMAIN_ITEMPASSED + +CLOSEDOMAIN_ITEMPASSED +~~~~~~~~~~~~~~~~~~~~~~ + +Default: ``0`` + +An integer which specifies a number of items. If the spider scrapes more than +that amount if items and those items are passed by the item pipeline, the +domain will be closed with the reason ``closedomain_itempassed``. If zero (or +non set) domains won't be closed by number of passed items. Stack trace dump extension --------------------------- diff --git a/docs/ref/settings.rst b/docs/ref/settings.rst index 09dfc5098..1aa387e88 100644 --- a/docs/ref/settings.rst +++ b/docs/ref/settings.rst @@ -81,28 +81,6 @@ Default: ``True`` Whether to split HTTP cache storage in several dirs for performance. -.. setting:: CLOSEDOMAIN_NOTIFY - -CLOSEDOMAIN_NOTIFY ------------------- - -Default: ``[]`` -Scope: ``scrapy.contrib.closedomain`` - -A list of emails to notify if the domain has been automatically closed by timeout. - -.. setting:: CLOSEDOMAIN_TIMEOUT - -CLOSEDOMAIN_TIMEOUT -------------------- - -Default: ``0`` -Scope: ``scrapy.contrib.closedomain`` - -A timeout (in secs) for automatically closing a spider. Spiders that remain -open for more than this time will be automatically closed. If zero, the -automatically closing is disabled. - .. setting:: CLUSTER_LOGDIR CLUSTER_LOGDIR diff --git a/scrapy/conf/default_settings.py b/scrapy/conf/default_settings.py index ae4a636f4..020fcb4c8 100644 --- a/scrapy/conf/default_settings.py +++ b/scrapy/conf/default_settings.py @@ -21,7 +21,7 @@ BOT_NAME = 'scrapybot' BOT_VERSION = '1.0' CLOSEDOMAIN_TIMEOUT = 0 -CLOSEDOMAIN_NOTIFY = [] +CLOSEDOMAIN_ITEMPASSED = 0 CLUSTER_LOGDIR = ''