18 KiB
Signals
Scrapy uses signals extensively to notify when certain events occur. You can catch some of those signals in your Scrapy project (using an :ref:`extension <topics-extensions>`, for example) to perform additional tasks or extend Scrapy to add functionality not provided out of the box.
System Message: ERROR/3 (<stdin>, line 7); backlink
Unknown interpreted text role "ref".Even though signals provide several arguments, the handlers that catch them don't need to accept all of them - the signal dispatching mechanism will only deliver the arguments that the handler receives.
You can connect to signals (or send your own) through the :ref:`topics-api-signals`.
System Message: ERROR/3 (<stdin>, line 16); backlink
Unknown interpreted text role "ref".Here is a simple example showing how you can catch signals and perform some action:
System Message: WARNING/2 (<stdin>, line 21)
Cannot analyze code. Pygments package not found.
.. code-block:: python
from scrapy import signals
from scrapy import Spider
class DmozSpider(Spider):
name = "dmoz"
allowed_domains = ["dmoz.org"]
start_urls = [
"http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
"http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/",
]
@classmethod
def from_crawler(cls, crawler, *args, **kwargs):
spider = super(DmozSpider, cls).from_crawler(crawler, *args, **kwargs)
crawler.signals.connect(spider.spider_closed, signal=signals.spider_closed)
return spider
def spider_closed(self, spider):
spider.logger.info("Spider closed: %s", spider.name)
def parse(self, response):
pass
Asynchronous signal handlers
Some signals support returning :class:`~twisted.internet.defer.Deferred` or :term:`awaitable objects <awaitable>` from their handlers, allowing you to run asynchronous code that does not block Scrapy. If a signal handler returns one of these objects, Scrapy waits for that asynchronous operation to finish.
System Message: ERROR/3 (<stdin>, line 52); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 52); backlink
Unknown interpreted text role "term".Let's take an example using :ref:`coroutines <topics-coroutines>`:
System Message: ERROR/3 (<stdin>, line 58); backlink
Unknown interpreted text role "ref".System Message: WARNING/2 (<stdin>, line 61)
Cannot analyze code. Pygments package not found.
.. code-block:: python
import scrapy
import treq
class SignalSpider(scrapy.Spider):
name = "signals"
start_urls = ["https://quotes.toscrape.com/page/1/"]
@classmethod
def from_crawler(cls, crawler, *args, **kwargs):
spider = super(SignalSpider, cls).from_crawler(crawler, *args, **kwargs)
crawler.signals.connect(spider.item_scraped, signal=signals.item_scraped)
return spider
async def item_scraped(self, item):
# Send the scraped item to the server
response = await treq.post(
"http://example.com/post",
json.dumps(item).encode("ascii"),
headers={b"Content-Type": [b"application/json"]},
)
return response
def parse(self, response):
for quote in response.css("div.quote"):
yield {
"text": quote.css("span.text::text").get(),
"author": quote.css("small.author::text").get(),
"tags": quote.css("div.tags a.tag::text").getall(),
}
See the :ref:`topics-signals-ref` below to know which signals support :class:`~twisted.internet.defer.Deferred` and :term:`awaitable objects <awaitable>`.
System Message: ERROR/3 (<stdin>, line 95); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 95); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 95); backlink
Unknown interpreted text role "term".Built-in signals reference
System Message: ERROR/3 (<stdin>, line 103)
Unknown directive type "module".
.. module:: scrapy.signals :synopsis: Signals definitions
Here's the list of Scrapy built-in signals and their meaning.
Engine signals
engine_started
System Message: ERROR/3 (<stdin>, line 115)
Unknown directive type "signal".
.. signal:: engine_started
System Message: ERROR/3 (<stdin>, line 116)
Unknown directive type "function".
.. function:: engine_started()
Sent when the Scrapy engine has started crawling.
This signal supports :ref:`asynchronous handlers <signal-deferred>`.
Note
This signal may be fired after the :signal:`spider_opened` signal, depending on how the spider was started. So don't rely on this signal getting fired before :signal:`spider_opened`.
System Message: ERROR/3 (<stdin>, line 122); backlink
Unknown interpreted text role "signal".System Message: ERROR/3 (<stdin>, line 122); backlink
Unknown interpreted text role "signal".engine_stopped
System Message: ERROR/3 (<stdin>, line 129)
Unknown directive type "signal".
.. signal:: engine_stopped
System Message: ERROR/3 (<stdin>, line 130)
Unknown directive type "function".
.. function:: engine_stopped()
Sent when the Scrapy engine is stopped (for example, when a crawling
process has finished).
This signal supports :ref:`asynchronous handlers <signal-deferred>`.
scheduler_empty
System Message: ERROR/3 (<stdin>, line 140)
Unknown directive type "signal".
.. signal:: scheduler_empty
System Message: ERROR/3 (<stdin>, line 141)
Unknown directive type "function".
.. function:: scheduler_empty()
Sent whenever the engine asks for a pending request from the
:ref:`scheduler <topics-scheduler>` (i.e. calls its
:meth:`~scrapy.core.scheduler.BaseScheduler.next_request` method) and the
scheduler returns none.
See :ref:`start-requests-lazy` for an example.
This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
Item signals
Note
As at max :setting:`CONCURRENT_ITEMS` items are processed in parallel, many deferreds are fired together using :class:`~twisted.internet.defer.DeferredList`. Hence the next batch waits for the :class:`~twisted.internet.defer.DeferredList` to fire and then runs the respective item signal handler for the next batch of scraped items.
System Message: ERROR/3 (<stdin>, line 157); backlink
Unknown interpreted text role "setting".System Message: ERROR/3 (<stdin>, line 157); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 157); backlink
Unknown interpreted text role "class".item_scraped
System Message: ERROR/3 (<stdin>, line 167)
Unknown directive type "signal".
.. signal:: item_scraped
System Message: ERROR/3 (<stdin>, line 168)
Unknown directive type "function".
.. function:: item_scraped(item, response, spider)
Sent when an item has been scraped, after it has passed all the
:ref:`topics-item-pipeline` stages (without being dropped).
This signal supports :ref:`asynchronous handlers <signal-deferred>`.
:param item: the scraped item
:type item: :ref:`item object <item-types>`
:param spider: the spider which scraped the item
:type spider: :class:`~scrapy.Spider` object
:param response: the response from where the item was scraped, or ``None``
if it was yielded from :meth:`~scrapy.Spider.start`.
:type response: :class:`~scrapy.http.Response` | ``None``
item_dropped
System Message: ERROR/3 (<stdin>, line 188)
Unknown directive type "signal".
.. signal:: item_dropped
System Message: ERROR/3 (<stdin>, line 189)
Unknown directive type "function".
.. function:: item_dropped(item, response, exception, spider)
Sent after an item has been dropped from the :ref:`topics-item-pipeline`
when some stage raised a :exc:`~scrapy.exceptions.DropItem` exception.
This signal supports :ref:`asynchronous handlers <signal-deferred>`.
:param item: the item dropped from the :ref:`topics-item-pipeline`
:type item: :ref:`item object <item-types>`
:param spider: the spider which scraped the item
:type spider: :class:`~scrapy.Spider` object
:param response: the response from where the item was dropped, or ``None``
if it was yielded from :meth:`~scrapy.Spider.start`.
:type response: :class:`~scrapy.http.Response` | ``None``
:param exception: the exception (which must be a
:exc:`~scrapy.exceptions.DropItem` subclass) which caused the item
to be dropped
:type exception: :exc:`~scrapy.exceptions.DropItem` exception
item_error
System Message: ERROR/3 (<stdin>, line 214)
Unknown directive type "signal".
.. signal:: item_error
System Message: ERROR/3 (<stdin>, line 215)
Unknown directive type "function".
.. function:: item_error(item, response, spider, failure)
Sent when a :ref:`topics-item-pipeline` generates an error (i.e. raises
an exception), except :exc:`~scrapy.exceptions.DropItem` exception.
This signal supports :ref:`asynchronous handlers <signal-deferred>`.
:param item: the item that caused the error in the :ref:`topics-item-pipeline`
:type item: :ref:`item object <item-types>`
:param response: the response being processed when the exception was
raised, or ``None`` if it was yielded from
:meth:`~scrapy.Spider.start`.
:type response: :class:`~scrapy.http.Response` | ``None``
:param spider: the spider which raised the exception
:type spider: :class:`~scrapy.Spider` object
:param failure: the exception raised
:type failure: twisted.python.failure.Failure
Spider signals
spider_closed
System Message: ERROR/3 (<stdin>, line 243)
Unknown directive type "signal".
.. signal:: spider_closed
System Message: ERROR/3 (<stdin>, line 244)
Unknown directive type "function".
.. function:: spider_closed(spider, reason)
Sent after a spider has been closed. This can be used to release per-spider
resources reserved on :signal:`spider_opened`.
This signal supports :ref:`asynchronous handlers <signal-deferred>`.
:param spider: the spider which has been closed
:type spider: :class:`~scrapy.Spider` object
:param reason: a string which describes the reason why the spider was closed. If
it was closed because the spider has completed scraping, the reason
is ``'finished'``. Otherwise, if the spider was manually closed by
calling the ``close_spider`` engine method, then the reason is the one
passed in the ``reason`` argument of that method (which defaults to
``'cancelled'``). If the engine was shutdown (for example, by hitting
Ctrl-C to stop it) the reason will be ``'shutdown'``.
:type reason: str
spider_opened
System Message: ERROR/3 (<stdin>, line 266)
Unknown directive type "signal".
.. signal:: spider_opened
System Message: ERROR/3 (<stdin>, line 267)
Unknown directive type "function".
.. function:: spider_opened(spider)
Sent after a spider has been opened for crawling. This is typically used to
reserve per-spider resources, but can be used for any task that needs to be
performed when a spider is opened.
This signal supports :ref:`asynchronous handlers <signal-deferred>`.
:param spider: the spider which has been opened
:type spider: :class:`~scrapy.Spider` object
spider_idle
System Message: ERROR/3 (<stdin>, line 281)
Unknown directive type "signal".
.. signal:: spider_idle
System Message: ERROR/3 (<stdin>, line 282)
Unknown directive type "function".
.. function:: spider_idle(spider)
Sent when a spider has gone idle, which means the spider has no further:
* requests waiting to be downloaded
* requests scheduled
* items being processed in the item pipeline
If the idle state persists after all handlers of this signal have finished,
the engine starts closing the spider. After the spider has finished
closing, the :signal:`spider_closed` signal is sent.
You may raise a :exc:`~scrapy.exceptions.DontCloseSpider` exception to
prevent the spider from being closed.
Alternatively, you may raise a :exc:`~scrapy.exceptions.CloseSpider`
exception to provide a custom spider closing reason. An
idle handler is the perfect place to put some code that assesses
the final spider results and update the final closing reason
accordingly (e.g. setting it to 'too_few_results' instead of
'finished').
This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
:param spider: the spider which has gone idle
:type spider: :class:`~scrapy.Spider` object
.. note:: Scheduling some requests in your :signal:`spider_idle` handler does
**not** guarantee that it can prevent the spider from being closed,
although it sometimes can. That's because the spider may still remain idle
if all the scheduled requests are rejected by the scheduler (e.g. filtered
due to duplication).
spider_error
System Message: ERROR/3 (<stdin>, line 318)
Unknown directive type "signal".
.. signal:: spider_error
System Message: ERROR/3 (<stdin>, line 319)
Unknown directive type "function".
.. function:: spider_error(failure, response, spider)
Sent when a spider callback generates an error (i.e. raises an exception).
This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
:param failure: the exception raised
:type failure: twisted.python.failure.Failure
:param response: the response being processed when the exception was raised
:type response: :class:`~scrapy.http.Response` object
:param spider: the spider which raised the exception
:type spider: :class:`~scrapy.Spider` object
feed_slot_closed
System Message: ERROR/3 (<stdin>, line 337)
Unknown directive type "signal".
.. signal:: feed_slot_closed
System Message: ERROR/3 (<stdin>, line 338)
Unknown directive type "function".
.. function:: feed_slot_closed(slot)
Sent when a :ref:`feed exports <topics-feed-exports>` slot is closed.
This signal supports :ref:`asynchronous handlers <signal-deferred>`.
:param slot: the slot closed
:type slot: scrapy.extensions.feedexport.FeedSlot
feed_exporter_closed
System Message: ERROR/3 (<stdin>, line 350)
Unknown directive type "signal".
.. signal:: feed_exporter_closed
System Message: ERROR/3 (<stdin>, line 351)
Unknown directive type "function".
.. function:: feed_exporter_closed()
Sent when the :ref:`feed exports <topics-feed-exports>` extension is closed,
during the handling of the :signal:`spider_closed` signal by the extension,
after all feed exporting has been handled.
This signal supports :ref:`asynchronous handlers <signal-deferred>`.
memusage_warning_reached
System Message: ERROR/3 (<stdin>, line 362)
Unknown directive type "signal".
.. signal:: memusage_warning_reached
System Message: ERROR/3 (<stdin>, line 364)
Unknown directive type "function".
.. function:: memusage_warning_reached()
Sent by the :class:`~scrapy.extensions.memusage.MemoryUsage` extension when the
memory usage reaches the warning threshold (:setting:`MEMUSAGE_WARNING_MB`).
This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
Request signals
request_scheduled
System Message: ERROR/3 (<stdin>, line 378)
Unknown directive type "signal".
.. signal:: request_scheduled
System Message: ERROR/3 (<stdin>, line 379)
Unknown directive type "function".
.. function:: request_scheduled(request, spider)
Sent when the engine is asked to schedule a :class:`~scrapy.Request`, to be
downloaded later, before the request reaches the :ref:`scheduler
<topics-scheduler>`.
Raise :exc:`~scrapy.exceptions.IgnoreRequest` to drop a request before it
reaches the scheduler.
This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
.. versionadded:: 2.11.2
Allow dropping requests with :exc:`~scrapy.exceptions.IgnoreRequest`.
:param request: the request that reached the scheduler
:type request: :class:`~scrapy.Request` object
:param spider: the spider that yielded the request
:type spider: :class:`~scrapy.Spider` object
request_dropped
System Message: ERROR/3 (<stdin>, line 402)
Unknown directive type "signal".
.. signal:: request_dropped
System Message: ERROR/3 (<stdin>, line 403)
Unknown directive type "function".
.. function:: request_dropped(request, spider)
Sent when a :class:`~scrapy.Request`, scheduled by the engine to be
downloaded later, is rejected by the scheduler.
This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
:param request: the request that reached the scheduler
:type request: :class:`~scrapy.Request` object
:param spider: the spider that yielded the request
:type spider: :class:`~scrapy.Spider` object
request_reached_downloader
System Message: ERROR/3 (<stdin>, line 419)
Unknown directive type "signal".
.. signal:: request_reached_downloader
System Message: ERROR/3 (<stdin>, line 420)
Unknown directive type "function".
.. function:: request_reached_downloader(request, spider)
Sent when a :class:`~scrapy.Request` reached downloader.
This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
:param request: the request that reached downloader
:type request: :class:`~scrapy.Request` object
:param spider: the spider that yielded the request
:type spider: :class:`~scrapy.Spider` object
request_left_downloader
System Message: ERROR/3 (<stdin>, line 435)
Unknown directive type "signal".
.. signal:: request_left_downloader
System Message: ERROR/3 (<stdin>, line 436)
Unknown directive type "function".
.. function:: request_left_downloader(request, spider)
Sent when a :class:`~scrapy.Request` leaves the downloader, even in case of
failure.
This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
:param request: the request that reached the downloader
:type request: :class:`~scrapy.Request` object
:param spider: the spider that yielded the request
:type spider: :class:`~scrapy.Spider` object
bytes_received
System Message: ERROR/3 (<stdin>, line 452)
Unknown directive type "signal".
.. signal:: bytes_received
System Message: ERROR/3 (<stdin>, line 453)
Unknown directive type "function".
.. function:: bytes_received(data, request, spider)
Sent by the HTTP 1.1 and S3 download handlers when a group of bytes is
received for a specific request. This signal might be fired multiple
times for the same request, with partial data each time. For instance,
a possible scenario for a 25 kb response would be two signals fired
with 10 kb of data, and a final one with 5 kb of data.
Handlers for this signal can stop the download of a response while it
is in progress by raising the :exc:`~scrapy.exceptions.StopDownload`
exception. Please refer to the :ref:`topics-stop-response-download` topic
for additional information and examples.
This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
:param data: the data received by the download handler
:type data: :class:`bytes` object
:param request: the request that generated the download
:type request: :class:`~scrapy.Request` object
:param spider: the spider associated with the response
:type spider: :class:`~scrapy.Spider` object
headers_received
System Message: ERROR/3 (<stdin>, line 480)
Unknown directive type "signal".
.. signal:: headers_received
System Message: ERROR/3 (<stdin>, line 481)
Unknown directive type "function".
.. function:: headers_received(headers, body_length, request, spider)
Sent by the HTTP 1.1 and S3 download handlers when the response headers are
available for a given request, before downloading any additional content.
Handlers for this signal can stop the download of a response while it
is in progress by raising the :exc:`~scrapy.exceptions.StopDownload`
exception. Please refer to the :ref:`topics-stop-response-download` topic
for additional information and examples.
This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
:param headers: the headers received by the download handler
:type headers: :class:`scrapy.http.headers.Headers` object
:param body_length: expected size of the response body, in bytes
:type body_length: `int`
:param request: the request that generated the download
:type request: :class:`~scrapy.Request` object
:param spider: the spider associated with the response
:type spider: :class:`~scrapy.Spider` object
Response signals
response_received
System Message: ERROR/3 (<stdin>, line 512)
Unknown directive type "signal".
.. signal:: response_received
System Message: ERROR/3 (<stdin>, line 513)
Unknown directive type "function".
.. function:: response_received(response, request, spider)
Sent when the engine receives a new :class:`~scrapy.http.Response` from the
downloader.
This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
:param response: the response received
:type response: :class:`~scrapy.http.Response` object
:param request: the request that generated the response
:type request: :class:`~scrapy.Request` object
:param spider: the spider for which the response is intended
:type spider: :class:`~scrapy.Spider` object
Note
The request argument might not contain the original request that reached the downloader, if a :ref:`topics-downloader-middleware` modifies the :class:`~scrapy.http.Response` object and sets a specific request attribute.
System Message: ERROR/3 (<stdin>, line 529); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 529); backlink
Unknown interpreted text role "class".response_downloaded
System Message: ERROR/3 (<stdin>, line 537)
Unknown directive type "signal".
.. signal:: response_downloaded
System Message: ERROR/3 (<stdin>, line 538)
Unknown directive type "function".
.. function:: response_downloaded(response, request, spider)
Sent by the downloader right after a :class:`~scrapy.http.Response` is downloaded.
This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
:param response: the response downloaded
:type response: :class:`~scrapy.http.Response` object
:param request: the request that generated the response
:type request: :class:`~scrapy.Request` object
:param spider: the spider for which the response is intended
:type spider: :class:`~scrapy.Spider` object