7.3 KiB

Raw Blame History

Signals

Scrapy uses signals extensively to notify when certain actions occur. You can catch some of those signals in your Scrapy project or extension to perform additional tasks or extend Scrapy to add functionality not provided out of the box.

Even though signals provide several arguments, the handlers which catch them don't have to receive all of them.

For more information about working when see the documentation of pydispatcher (library used to implement signals).

Built-in signals reference

System Message: ERROR/3 (<stdin>, line 26)

Unknown directive type "module".

.. module:: scrapy.core.signals
   :synopsis: Signals definitions

Here's a list of signals used in Scrapy and their meaning, in alphabetical order.

domain_closed

System Message: ERROR/3 (<stdin>, line 35)

Unknown directive type "signal".

.. signal:: domain_closed

System Message: ERROR/3 (<stdin>, line 36)

Unknown directive type "function".

.. function:: domain_closed(domain, spider, reason)

    Sent after a spider/domain has been closed. This can be used to release
    per-spider resources reserved on :signal:`domain_opened`.

    :param domain: a string which contains the domain of the spider which has
        been closed
    :type domain: str

    :param spider: the spider which has been closed
    :type spider: :class:`~scrapy.spider.BaseSpider` object

    :param reason: a string which describes the reason why the domain was closed. If
        it was closed because the domain has completed scraping, it the reason
        is ``'finished'``. Otherwise, if the domain was manually closed by
        calling the ``close_domain`` engine method, then the reason is the one
        passed in the ``reason`` argument of that method (which defaults to
        ``'cancelled'``). If the engine was shutdown (for example, by hitting
        Ctrl-C to stop it) the reason will be ``'shutdown'``.
    :type reason: str

domain_opened

System Message: ERROR/3 (<stdin>, line 60)

Unknown directive type "signal".

.. signal:: domain_opened

System Message: ERROR/3 (<stdin>, line 61)

Unknown directive type "function".

.. function:: domain_opened(domain, spider)

    Sent after a spider/domain has been opened for crawling. This is typically
    used to reserve per-spider resources, but can be used for any task that
    needs to be performed when a spider/domain is opened.

    :param domain: a string with the domain of the spider which has been opened
    :type domain: str

    :param spider: the spider which has been opened
    :type spider: :class:`~scrapy.spider.BaseSpider` object

domain_idle

System Message: ERROR/3 (<stdin>, line 76)

Unknown directive type "signal".

.. signal:: domain_idle

System Message: ERROR/3 (<stdin>, line 77)

Unknown directive type "function".

.. function:: domain_idle(domain, spider)

    Sent when a domain has gone idle, which means the spider has no further:

        * requests waiting to be downloaded
        * requests scheduled
        * items being processed in the item pipeline

    If the idle state persists after all handlers of this signal have finished,
    the engine starts closing the domain. After the domain has finished
    closing, the :signal:`domain_closed` signal is sent.

    You can, for example, schedule some requests in your :signal:`domain_idle`
    handler to prevent the domain from being closed.

    :param domain: is a string with the domain of the spider which has gone idle
    :type domain: str

    :param spider: the spider which has gone idle
    :type spider: :class:`~scrapy.spider.BaseSpider` object

engine_started

System Message: ERROR/3 (<stdin>, line 102)

Unknown directive type "signal".

.. signal:: engine_started

System Message: ERROR/3 (<stdin>, line 103)

Unknown directive type "function".

.. function:: engine_started()

    Sent when the Scrapy engine is started (for example, when a crawling
    process has started).

engine_stopped

System Message: ERROR/3 (<stdin>, line 111)

Unknown directive type "signal".

.. signal:: engine_stopped

System Message: ERROR/3 (<stdin>, line 112)

Unknown directive type "function".

.. function:: engine_stopped()

    Sent when the Scrapy engine is stopped (for example, when a crawling
    process has finished).

item_scraped

System Message: ERROR/3 (<stdin>, line 120)

Unknown directive type "signal".

.. signal:: item_scraped

System Message: ERROR/3 (<stdin>, line 121)

Unknown directive type "function".

.. function:: item_scraped(item, spider, response)

    Sent when the engine receives a new scraped item from the spider, and right
    before the item is sent to the :ref:`topics-item-pipeline`.

    :param item: is the item scraped
    :type item: :class:`~scrapy.item.Item` object

    :param spider: the spider which scraped the item
    :type spider: :class:`~scrapy.spider.BaseSpider` object

    :param response: the response from which the item was scraped
    :type response: :class:`~scrapy.http.Response` object

item_passed

System Message: ERROR/3 (<stdin>, line 138)

Unknown directive type "signal".

.. signal:: item_passed

System Message: ERROR/3 (<stdin>, line 139)

Unknown directive type "function".

.. function:: item_passed(item, spider, output)

    Sent after an item has passed all the :ref:`topics-item-pipeline` stages without
    being dropped.

    :param item: the item which passed the pipeline
    :type item: :class:`~scrapy.item.Item` object

    :param spider: the spider which scraped the item
    :type spider: :class:`~scrapy.spider.BaseSpider` object

    :param output: the output of the item pipeline. This is typically the
        same :class:`~scrapy.item.Item` object received in the ``item``
        parameter, unless some pipeline stage created a new item.

item_dropped

System Message: ERROR/3 (<stdin>, line 157)

Unknown directive type "signal".

.. signal:: item_dropped

System Message: ERROR/3 (<stdin>, line 158)

Unknown directive type "function".

.. function:: item_dropped(item, spider, exception)

    Sent after an item has been dropped from the :ref:`topics-item-pipeline`
    when some stage raised a :exc:`~scrapy.core.exceptions.DropItem` exception.

    :param item: the item dropped from the :ref:`topics-item-pipeline`
    :type item: :class:`~scrapy.item.Item` object

    :param spider: the spider which scraped the item
    :type spider: :class:`~scrapy.spider.BaseSpider` object

    :param exception: the exception (which must be a
        :exc:`~scrapy.core.exceptions.DropItem` subclass) which caused the item
        to be dropped
    :type exception: :exc:`~scrapy.core.exceptions.DropItem` exception

request_received

System Message: ERROR/3 (<stdin>, line 177)

Unknown directive type "signal".

.. signal:: request_received

System Message: ERROR/3 (<stdin>, line 178)

Unknown directive type "function".

.. function:: request_received(request, spider, response)

    Sent when the engine receives a :class:`~scrapy.http.Request` from a spider.

    :param request: the request received
    :type request: :class:`~scrapy.http.Request` object

    :param spider: the spider which generated the request
    :type spider: :class:`~scrapy.spider.BaseSpider` object

    :param response: the :class:`~scrapy.http.Response` fed to the spider which
        generated the request later
    :type response: :class:`~scrapy.http.Response` object

request_uploaded

System Message: ERROR/3 (<stdin>, line 195)

Unknown directive type "signal".

.. signal:: request_uploaded

System Message: ERROR/3 (<stdin>, line 196)

Unknown directive type "function".

.. function:: request_uploaded(request, spider)

    Sent right after the download has sent a :class:`~scrapy.http.Request`.

    :param request: the request uploaded/sent
    :type request: :class:`~scrapy.http.Request` object

    :param spider: the spider which generated the request
    :type spider: :class:`~scrapy.spider.BaseSpider` object

response_received

System Message: ERROR/3 (<stdin>, line 209)

Unknown directive type "signal".

.. signal:: response_received

System Message: ERROR/3 (<stdin>, line 210)

Unknown directive type "function".

.. function:: response_received(response, spider)

    :param response: the response received
    :type response: :class:`~scrapy.http.Response` object

    :param spider: the spider for which the response is intended
    :type spider: :class:`~scrapy.spider.BaseSpider` object

    Sent when the engine receives a new :class:`~scrapy.http.Response` from the
    downloader.

response_downloaded

System Message: ERROR/3 (<stdin>, line 224)

Unknown directive type "signal".

.. signal:: response_downloaded

System Message: ERROR/3 (<stdin>, line 225)

Unknown directive type "function".

.. function:: response_downloaded(response, spider)

    Sent by the downloader right after a ``HTTPResponse`` is downloaded.

    :param response: the response downloaded
    :type response: :class:`~scrapy.http.Response` object

    :param spider: the spider for which the response is intended
    :type spider: :class:`~scrapy.spider.BaseSpider` object

</html>

7.3 KiB Raw Blame History

Signals

Built-in signals reference

domain_closed

domain_opened

domain_idle

engine_started

engine_stopped

item_scraped

item_passed

item_dropped

request_received

request_uploaded

response_received

response_downloaded

7.3 KiB

Raw Blame History