diff --git a/docs/topics/signals.rst b/docs/topics/signals.rst index 59742ffeb..a815ffb43 100644 --- a/docs/topics/signals.rst +++ b/docs/topics/signals.rst @@ -60,6 +60,7 @@ Let's take an example using :ref:`coroutines `: .. code-block:: python import scrapy + import treq class SignalSpider(scrapy.Spider): @@ -103,6 +104,7 @@ Built-in signals reference Here's the list of Scrapy built-in signals and their meaning. + Engine signals -------------- @@ -114,7 +116,7 @@ engine_started Sent when the Scrapy engine has started crawling. - This signal supports asynchronous handlers. + This signal supports :ref:`asynchronous handlers `. .. note:: This signal may be fired *after* the :signal:`spider_opened` signal, depending on how the spider was started. So **don't** rely on this signal @@ -129,7 +131,7 @@ engine_stopped Sent when the Scrapy engine is stopped (for example, when a crawling process has finished). - This signal supports asynchronous handlers. + This signal supports :ref:`asynchronous handlers `. scheduler_empty ~~~~~~~~~~~~~~~ @@ -144,6 +146,9 @@ scheduler_empty See :ref:`start-requests-lazy` for an example. + This signal does not support :ref:`asynchronous handlers `. + + Item signals ------------ @@ -164,7 +169,7 @@ item_scraped Sent when an item has been scraped, after it has passed all the :ref:`topics-item-pipeline` stages (without being dropped). - This signal supports asynchronous handlers. + This signal supports :ref:`asynchronous handlers `. :param item: the scraped item :type item: :ref:`item object ` @@ -185,7 +190,7 @@ item_dropped Sent after an item has been dropped from the :ref:`topics-item-pipeline` when some stage raised a :exc:`~scrapy.exceptions.DropItem` exception. - This signal supports asynchronous handlers. + This signal supports :ref:`asynchronous handlers `. :param item: the item dropped from the :ref:`topics-item-pipeline` :type item: :ref:`item object ` @@ -211,7 +216,7 @@ item_error Sent when a :ref:`topics-item-pipeline` generates an error (i.e. raises an exception), except :exc:`~scrapy.exceptions.DropItem` exception. - This signal supports asynchronous handlers. + This signal supports :ref:`asynchronous handlers `. :param item: the item that caused the error in the :ref:`topics-item-pipeline` :type item: :ref:`item object ` @@ -227,6 +232,7 @@ item_error :param failure: the exception raised :type failure: twisted.python.failure.Failure + Spider signals -------------- @@ -239,7 +245,7 @@ spider_closed Sent after a spider has been closed. This can be used to release per-spider resources reserved on :signal:`spider_opened`. - This signal supports asynchronous handlers. + This signal supports :ref:`asynchronous handlers `. :param spider: the spider which has been closed :type spider: :class:`~scrapy.Spider` object @@ -263,7 +269,7 @@ spider_opened reserve per-spider resources, but can be used for any task that needs to be performed when a spider is opened. - This signal supports asynchronous handlers. + This signal supports :ref:`asynchronous handlers `. :param spider: the spider which has been opened :type spider: :class:`~scrapy.Spider` object @@ -294,16 +300,16 @@ spider_idle accordingly (e.g. setting it to 'too_few_results' instead of 'finished'). - This signal does not support returning deferreds from its handlers. + This signal does not support :ref:`asynchronous handlers `. :param spider: the spider which has gone idle :type spider: :class:`~scrapy.Spider` object -.. note:: Scheduling some requests in your :signal:`spider_idle` handler does - **not** guarantee that it can prevent the spider from being closed, - although it sometimes can. That's because the spider may still remain idle - if all the scheduled requests are rejected by the scheduler (e.g. filtered - due to duplication). + .. note:: Scheduling some requests in your :signal:`spider_idle` handler does + **not** guarantee that it can prevent the spider from being closed, + although it sometimes can. That's because the spider may still remain idle + if all the scheduled requests are rejected by the scheduler (e.g. filtered + due to duplication). spider_error ~~~~~~~~~~~~ @@ -313,7 +319,7 @@ spider_error Sent when a spider callback generates an error (i.e. raises an exception). - This signal does not support returning deferreds from its handlers. + This signal does not support :ref:`asynchronous handlers `. :param failure: the exception raised :type failure: twisted.python.failure.Failure @@ -332,12 +338,11 @@ feed_slot_closed Sent when a :ref:`feed exports ` slot is closed. - This signal supports asynchronous handlers. + This signal supports :ref:`asynchronous handlers `. :param slot: the slot closed :type slot: scrapy.extensions.feedexport.FeedSlot - feed_exporter_closed ~~~~~~~~~~~~~~~~~~~~ @@ -348,7 +353,7 @@ feed_exporter_closed during the handling of the :signal:`spider_closed` signal by the extension, after all feed exporting has been handled. - This signal supports asynchronous handlers. + This signal supports :ref:`asynchronous handlers `. Request signals @@ -367,7 +372,7 @@ request_scheduled Raise :exc:`~scrapy.exceptions.IgnoreRequest` to drop a request before it reaches the scheduler. - This signal does not support returning deferreds from its handlers. + This signal does not support :ref:`asynchronous handlers `. .. versionadded:: 2.11.2 Allow dropping requests with :exc:`~scrapy.exceptions.IgnoreRequest`. @@ -387,7 +392,7 @@ request_dropped Sent when a :class:`~scrapy.Request`, scheduled by the engine to be downloaded later, is rejected by the scheduler. - This signal does not support returning deferreds from its handlers. + This signal does not support :ref:`asynchronous handlers `. :param request: the request that reached the scheduler :type request: :class:`~scrapy.Request` object @@ -403,7 +408,7 @@ request_reached_downloader Sent when a :class:`~scrapy.Request` reached downloader. - This signal does not support returning deferreds from its handlers. + This signal does not support :ref:`asynchronous handlers `. :param request: the request that reached downloader :type request: :class:`~scrapy.Request` object @@ -422,7 +427,7 @@ request_left_downloader Sent when a :class:`~scrapy.Request` leaves the downloader, even in case of failure. - This signal does not support returning deferreds from its handlers. + This signal does not support :ref:`asynchronous handlers `. :param request: the request that reached the downloader :type request: :class:`~scrapy.Request` object @@ -433,11 +438,11 @@ request_left_downloader bytes_received ~~~~~~~~~~~~~~ -.. versionadded:: 2.2 - .. signal:: bytes_received .. function:: bytes_received(data, request, spider) + .. versionadded:: 2.2 + Sent by the HTTP 1.1 and S3 download handlers when a group of bytes is received for a specific request. This signal might be fired multiple times for the same request, with partial data each time. For instance, @@ -449,7 +454,7 @@ bytes_received exception. Please refer to the :ref:`topics-stop-response-download` topic for additional information and examples. - This signal does not support returning deferreds from its handlers. + This signal does not support :ref:`asynchronous handlers `. :param data: the data received by the download handler :type data: :class:`bytes` object @@ -463,11 +468,11 @@ bytes_received headers_received ~~~~~~~~~~~~~~~~ -.. versionadded:: 2.5 - .. signal:: headers_received .. function:: headers_received(headers, body_length, request, spider) + .. versionadded:: 2.5 + Sent by the HTTP 1.1 and S3 download handlers when the response headers are available for a given request, before downloading any additional content. @@ -476,7 +481,7 @@ headers_received exception. Please refer to the :ref:`topics-stop-response-download` topic for additional information and examples. - This signal does not support returning deferreds from its handlers. + This signal does not support :ref:`asynchronous handlers `. :param headers: the headers received by the download handler :type headers: :class:`scrapy.http.headers.Headers` object @@ -490,6 +495,7 @@ headers_received :param spider: the spider associated with the response :type spider: :class:`~scrapy.Spider` object + Response signals ---------------- @@ -502,7 +508,7 @@ response_received Sent when the engine receives a new :class:`~scrapy.http.Response` from the downloader. - This signal does not support returning deferreds from its handlers. + This signal does not support :ref:`asynchronous handlers `. :param response: the response received :type response: :class:`~scrapy.http.Response` object @@ -524,9 +530,9 @@ response_downloaded .. signal:: response_downloaded .. function:: response_downloaded(response, request, spider) - Sent by the downloader right after a ``HTTPResponse`` is downloaded. + Sent by the downloader right after a :class:`~scrapy.http.Response` is downloaded. - This signal does not support returning deferreds from its handlers. + This signal does not support :ref:`asynchronous handlers `. :param response: the response downloaded :type response: :class:`~scrapy.http.Response` object diff --git a/scrapy/signalmanager.py b/scrapy/signalmanager.py index 7fd172535..283060074 100644 --- a/scrapy/signalmanager.py +++ b/scrapy/signalmanager.py @@ -53,7 +53,8 @@ class SignalManager: self, signal: Any, **kwargs: Any ) -> Deferred[list[tuple[Any, Any]]]: """ - Like :meth:`send_catch_log` but supports asynchronous signal handlers. + Like :meth:`send_catch_log` but supports :ref:`asynchronous signal + handlers `. Returns a Deferred that gets fired once all signal handlers have finished. Send a signal, catch exceptions and log them. @@ -68,7 +69,8 @@ class SignalManager: self, signal: Any, **kwargs: Any ) -> list[tuple[Any, Any]]: """ - Like :meth:`send_catch_log` but supports asynchronous signal handlers. + Like :meth:`send_catch_log` but supports :ref:`asynchronous signal + handlers `. Returns a coroutine that completes once all signal handlers have finished. Send a signal, catch exceptions and log them. diff --git a/scrapy/utils/signal.py b/scrapy/utils/signal.py index d6b0a671b..552fbaa90 100644 --- a/scrapy/utils/signal.py +++ b/scrapy/utils/signal.py @@ -30,7 +30,7 @@ def send_catch_log( *arguments: TypingAny, **named: TypingAny, ) -> list[tuple[TypingAny, TypingAny]]: - """Like pydispatcher.robust.sendRobust but it also logs errors and returns + """Like ``pydispatcher.robust.sendRobust()`` but it also logs errors and returns Failures instead of exceptions. """ dont_log = named.pop("dont_log", ()) @@ -73,7 +73,8 @@ def send_catch_log_deferred( *arguments: TypingAny, **named: TypingAny, ) -> Generator[Deferred[TypingAny], TypingAny, list[tuple[TypingAny, TypingAny]]]: - """Like send_catch_log but supports asynchronous signal handlers. + """Like :func:`send_catch_log` but supports :ref:`asynchronous signal handlers + `. Returns a deferred that gets fired once all signal handlers have finished. """ @@ -115,7 +116,8 @@ async def send_catch_log_async( *arguments: TypingAny, **named: TypingAny, ) -> list[tuple[TypingAny, TypingAny]]: - """Like send_catch_log but supports asynchronous signal handlers. + """Like :func:`send_catch_log` but supports :ref:`asynchronous signal handlers + `. Returns a coroutine that completes once all signal handlers have finished. """ @@ -126,7 +128,7 @@ async def send_catch_log_async( def disconnect_all(signal: TypingAny = Any, sender: TypingAny = Any) -> None: """Disconnect all signal handlers. Useful for cleaning up after running - tests + tests. """ for receiver in liveReceivers(getAllReceivers(sender, signal)): disconnect(receiver, signal=signal, sender=sender)