RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Slight improvements for the signal docs.

This commit is contained in:
Andrey Rakhmatullin 2025-05-15 20:02:38 +05:00
parent 1ddcb568e2
commit bf1bfaaa3e
3 changed files with 46 additions and 36 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

66
docs/topics/signals.rst
View File

@ -60,6 +60,7 @@ Let's take an example using :ref:`coroutines <topics-coroutines>`:
.. code-block:: python
import scrapy
import treq
class SignalSpider(scrapy.Spider):
@ -103,6 +104,7 @@ Built-in signals reference
Here's the list of Scrapy built-in signals and their meaning.
Engine signals
--------------
@ -114,7 +116,7 @@ engine_started
Sent when the Scrapy engine has started crawling.
This signal supports asynchronous handlers.
This signal supports :ref:`asynchronous handlers <signal-deferred>`.
.. note:: This signal may be fired *after* the :signal:`spider_opened` signal,
depending on how the spider was started. So **don't** rely on this signal
@ -129,7 +131,7 @@ engine_stopped
Sent when the Scrapy engine is stopped (for example, when a crawling
process has finished).
This signal supports asynchronous handlers.
This signal supports :ref:`asynchronous handlers <signal-deferred>`.
scheduler_empty
~~~~~~~~~~~~~~~
@ -144,6 +146,9 @@ scheduler_empty
See :ref:`start-requests-lazy` for an example.
This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
Item signals
------------
@ -164,7 +169,7 @@ item_scraped
Sent when an item has been scraped, after it has passed all the
:ref:`topics-item-pipeline` stages (without being dropped).
This signal supports asynchronous handlers.
This signal supports :ref:`asynchronous handlers <signal-deferred>`.
:param item: the scraped item
:type item: :ref:`item object <item-types>`
@ -185,7 +190,7 @@ item_dropped
Sent after an item has been dropped from the :ref:`topics-item-pipeline`
when some stage raised a :exc:`~scrapy.exceptions.DropItem` exception.
This signal supports asynchronous handlers.
This signal supports :ref:`asynchronous handlers <signal-deferred>`.
:param item: the item dropped from the :ref:`topics-item-pipeline`
:type item: :ref:`item object <item-types>`
@ -211,7 +216,7 @@ item_error
Sent when a :ref:`topics-item-pipeline` generates an error (i.e. raises
an exception), except :exc:`~scrapy.exceptions.DropItem` exception.
This signal supports asynchronous handlers.
This signal supports :ref:`asynchronous handlers <signal-deferred>`.
:param item: the item that caused the error in the :ref:`topics-item-pipeline`
:type item: :ref:`item object <item-types>`
@ -227,6 +232,7 @@ item_error
:param failure: the exception raised
:type failure: twisted.python.failure.Failure
Spider signals
--------------
@ -239,7 +245,7 @@ spider_closed
Sent after a spider has been closed. This can be used to release per-spider
resources reserved on :signal:`spider_opened`.
This signal supports asynchronous handlers.
This signal supports :ref:`asynchronous handlers <signal-deferred>`.
:param spider: the spider which has been closed
:type spider: :class:`~scrapy.Spider` object
@ -263,7 +269,7 @@ spider_opened
reserve per-spider resources, but can be used for any task that needs to be
performed when a spider is opened.
This signal supports asynchronous handlers.
This signal supports :ref:`asynchronous handlers <signal-deferred>`.
:param spider: the spider which has been opened
:type spider: :class:`~scrapy.Spider` object
@ -294,16 +300,16 @@ spider_idle
accordingly (e.g. setting it to 'too_few_results' instead of
'finished').
This signal does not support returning deferreds from its handlers.
This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
:param spider: the spider which has gone idle
:type spider: :class:`~scrapy.Spider` object
.. note:: Scheduling some requests in your :signal:`spider_idle` handler does
**not** guarantee that it can prevent the spider from being closed,
although it sometimes can. That's because the spider may still remain idle
if all the scheduled requests are rejected by the scheduler (e.g. filtered
due to duplication).
.. note:: Scheduling some requests in your :signal:`spider_idle` handler does
**not** guarantee that it can prevent the spider from being closed,
although it sometimes can. That's because the spider may still remain idle
if all the scheduled requests are rejected by the scheduler (e.g. filtered
due to duplication).
spider_error
~~~~~~~~~~~~
@ -313,7 +319,7 @@ spider_error
Sent when a spider callback generates an error (i.e. raises an exception).
This signal does not support returning deferreds from its handlers.
This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
:param failure: the exception raised
:type failure: twisted.python.failure.Failure
@ -332,12 +338,11 @@ feed_slot_closed
Sent when a :ref:`feed exports <topics-feed-exports>` slot is closed.
This signal supports asynchronous handlers.
This signal supports :ref:`asynchronous handlers <signal-deferred>`.
:param slot: the slot closed
:type slot: scrapy.extensions.feedexport.FeedSlot
feed_exporter_closed
~~~~~~~~~~~~~~~~~~~~
@ -348,7 +353,7 @@ feed_exporter_closed
during the handling of the :signal:`spider_closed` signal by the extension,
after all feed exporting has been handled.
This signal supports asynchronous handlers.
This signal supports :ref:`asynchronous handlers <signal-deferred>`.
Request signals
@ -367,7 +372,7 @@ request_scheduled
Raise :exc:`~scrapy.exceptions.IgnoreRequest` to drop a request before it
reaches the scheduler.
This signal does not support returning deferreds from its handlers.
This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
.. versionadded:: 2.11.2
Allow dropping requests with :exc:`~scrapy.exceptions.IgnoreRequest`.
@ -387,7 +392,7 @@ request_dropped
Sent when a :class:`~scrapy.Request`, scheduled by the engine to be
downloaded later, is rejected by the scheduler.
This signal does not support returning deferreds from its handlers.
This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
:param request: the request that reached the scheduler
:type request: :class:`~scrapy.Request` object
@ -403,7 +408,7 @@ request_reached_downloader
Sent when a :class:`~scrapy.Request` reached downloader.
This signal does not support returning deferreds from its handlers.
This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
:param request: the request that reached downloader
:type request: :class:`~scrapy.Request` object
@ -422,7 +427,7 @@ request_left_downloader
Sent when a :class:`~scrapy.Request` leaves the downloader, even in case of
failure.
This signal does not support returning deferreds from its handlers.
This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
:param request: the request that reached the downloader
:type request: :class:`~scrapy.Request` object
@ -433,11 +438,11 @@ request_left_downloader
bytes_received
~~~~~~~~~~~~~~
.. versionadded:: 2.2
.. signal:: bytes_received
.. function:: bytes_received(data, request, spider)
.. versionadded:: 2.2
Sent by the HTTP 1.1 and S3 download handlers when a group of bytes is
received for a specific request. This signal might be fired multiple
times for the same request, with partial data each time. For instance,
@ -449,7 +454,7 @@ bytes_received
exception. Please refer to the :ref:`topics-stop-response-download` topic
for additional information and examples.
This signal does not support returning deferreds from its handlers.
This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
:param data: the data received by the download handler
:type data: :class:`bytes` object
@ -463,11 +468,11 @@ bytes_received
headers_received
~~~~~~~~~~~~~~~~
.. versionadded:: 2.5
.. signal:: headers_received
.. function:: headers_received(headers, body_length, request, spider)
.. versionadded:: 2.5
Sent by the HTTP 1.1 and S3 download handlers when the response headers are
available for a given request, before downloading any additional content.
@ -476,7 +481,7 @@ headers_received
exception. Please refer to the :ref:`topics-stop-response-download` topic
for additional information and examples.
This signal does not support returning deferreds from its handlers.
This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
:param headers: the headers received by the download handler
:type headers: :class:`scrapy.http.headers.Headers` object
@ -490,6 +495,7 @@ headers_received
:param spider: the spider associated with the response
:type spider: :class:`~scrapy.Spider` object
Response signals
----------------
@ -502,7 +508,7 @@ response_received
Sent when the engine receives a new :class:`~scrapy.http.Response` from the
downloader.
This signal does not support returning deferreds from its handlers.
This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
:param response: the response received
:type response: :class:`~scrapy.http.Response` object
@ -524,9 +530,9 @@ response_downloaded
.. signal:: response_downloaded
.. function:: response_downloaded(response, request, spider)
Sent by the downloader right after a ``HTTPResponse`` is downloaded.
Sent by the downloader right after a :class:`~scrapy.http.Response` is downloaded.
This signal does not support returning deferreds from its handlers.
This signal does not support :ref:`asynchronous handlers <signal-deferred>`.
:param response: the response downloaded
:type response: :class:`~scrapy.http.Response` object

6
scrapy/signalmanager.py
View File

@ -53,7 +53,8 @@ class SignalManager:
self, signal: Any, **kwargs: Any
) -> Deferred[list[tuple[Any, Any]]]:
"""
Like :meth:`send_catch_log` but supports asynchronous signal handlers.
Like :meth:`send_catch_log` but supports :ref:`asynchronous signal
handlers <signal-deferred>`.
Returns a Deferred that gets fired once all signal handlers
have finished. Send a signal, catch exceptions and log them.
@ -68,7 +69,8 @@ class SignalManager:
self, signal: Any, **kwargs: Any
) -> list[tuple[Any, Any]]:
"""
Like :meth:`send_catch_log` but supports asynchronous signal handlers.
Like :meth:`send_catch_log` but supports :ref:`asynchronous signal
handlers <signal-deferred>`.
Returns a coroutine that completes once all signal handlers
have finished. Send a signal, catch exceptions and log them.

10
scrapy/utils/signal.py
View File

@ -30,7 +30,7 @@ def send_catch_log(
*arguments: TypingAny,
**named: TypingAny,
) -> list[tuple[TypingAny, TypingAny]]:
"""Like pydispatcher.robust.sendRobust but it also logs errors and returns
"""Like ``pydispatcher.robust.sendRobust()`` but it also logs errors and returns
Failures instead of exceptions.
"""
dont_log = named.pop("dont_log", ())
@ -73,7 +73,8 @@ def send_catch_log_deferred(
*arguments: TypingAny,
**named: TypingAny,
) -> Generator[Deferred[TypingAny], TypingAny, list[tuple[TypingAny, TypingAny]]]:
"""Like send_catch_log but supports asynchronous signal handlers.
"""Like :func:`send_catch_log` but supports :ref:`asynchronous signal handlers
<signal-deferred>`.
Returns a deferred that gets fired once all signal handlers have finished.
"""
@ -115,7 +116,8 @@ async def send_catch_log_async(
*arguments: TypingAny,
**named: TypingAny,
) -> list[tuple[TypingAny, TypingAny]]:
"""Like send_catch_log but supports asynchronous signal handlers.
"""Like :func:`send_catch_log` but supports :ref:`asynchronous signal handlers
<signal-deferred>`.
Returns a coroutine that completes once all signal handlers have finished.
"""
@ -126,7 +128,7 @@ async def send_catch_log_async(
def disconnect_all(signal: TypingAny = Any, sender: TypingAny = Any) -> None:
"""Disconnect all signal handlers. Useful for cleaning up after running
tests
tests.
"""
for receiver in liveReceivers(getAllReceivers(sender, signal)):
disconnect(receiver, signal=signal, sender=sender)