From f712513ed709e2e6d71a19b4e259934fd6f86955 Mon Sep 17 00:00:00 2001 From: Pengyu CHEN Date: Thu, 15 Jun 2017 10:41:02 +0800 Subject: [PATCH] Added doc for `scrapy.exceptions.DontCloseSpider`. Also fixes inaccurate doc for `scrapy.signals.spider_idle`. --- docs/topics/exceptions.rst | 8 ++++++++ docs/topics/signals.rst | 10 ++++++++-- 2 files changed, 16 insertions(+), 2 deletions(-) diff --git a/docs/topics/exceptions.rst b/docs/topics/exceptions.rst index cc02369d4..09cb8ed66 100644 --- a/docs/topics/exceptions.rst +++ b/docs/topics/exceptions.rst @@ -39,6 +39,14 @@ For example:: if 'Bandwidth exceeded' in response.body: raise CloseSpider('bandwidth_exceeded') +DontCloseSpider +--------------- + +.. exception:: DontCloseSpider + +This exception can be raised in a :signal:`spider_idle` signal handler to +prevent the spider from being closed. + IgnoreRequest ------------- diff --git a/docs/topics/signals.rst b/docs/topics/signals.rst index 0306ee4a5..cf1588df8 100644 --- a/docs/topics/signals.rst +++ b/docs/topics/signals.rst @@ -189,14 +189,20 @@ spider_idle the engine starts closing the spider. After the spider has finished closing, the :signal:`spider_closed` signal is sent. - You can, for example, schedule some requests in your :signal:`spider_idle` - handler to prevent the spider from being closed. + You may raise a :exc:`~scrapy.exceptions.DontCloseSpider` exception to + prevent the spider from being closed. This signal does not support returning deferreds from their handlers. :param spider: the spider which has gone idle :type spider: :class:`~scrapy.spiders.Spider` object +.. note:: Scheduling some requests in your :signal:`spider_idle` handler does + **not** guarantee that it can prevent the spider from being closed, + although it sometimes can. That's because the spider may still remain idle + if all the scheduled requests are rejected by the scheduler (e.g. filtered + due to duplication). + spider_error ------------