From a97dc55c714d90378dbc8a819cae1a3a40056d6e Mon Sep 17 00:00:00 2001
From: Andrey Rakhmatullin <wrar@wrar.name>
Date: Thu, 25 Mar 2021 17:37:32 +0500
Subject: [PATCH] Add/improve docs.

---
 docs/topics/asyncio.rst           |  1 -
 docs/topics/coroutines.rst        | 13 ++++++++++++-
 docs/topics/spider-middleware.rst | 17 +++++++++++------
 3 files changed, 23 insertions(+), 8 deletions(-)

diff --git a/docs/topics/asyncio.rst b/docs/topics/asyncio.rst
index 4addaa178..18712c928 100644
--- a/docs/topics/asyncio.rst
+++ b/docs/topics/asyncio.rst
@@ -38,7 +38,6 @@ You can also use custom asyncio event loops with the asyncio reactor. Set the
 :setting:`ASYNCIO_EVENT_LOOP` setting to the import path of the desired event loop class to
 use it instead of the default asyncio event loop.
 
-
 .. _asyncio-await-dfd:
 
 Awaiting on Deferreds
diff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rst
index 279632653..9b50d9312 100644
--- a/docs/topics/coroutines.rst
+++ b/docs/topics/coroutines.rst
@@ -17,6 +17,10 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):
 
 -   :class:`~scrapy.http.Request` callbacks.
 
+    .. versionchanged:: VERSION
+       Output of async callbacks is now processed asynchronously instead of collecting
+       all of it first.
+
 -   The :meth:`process_item` method of
     :ref:`item pipelines <topics-item-pipeline>`.
 
@@ -30,6 +34,13 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):
 
 -   :ref:`Signal handlers that support deferreds <signal-deferred>`.
 
+-   The :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output`
+    method of :ref:`spider middlewares <custom-spider-middleware>`.
+
+    .. versionadded:: VERSION
+    .. note:: This method needs to be an async generator, not just a coroutine that
+              returns an iterable.
+
 Usage
 =====
 
@@ -76,7 +87,7 @@ This means you can use many useful Python libraries providing such code::
         async def parse_with_asyncio(self, response):
             async with aiohttp.ClientSession() as session:
                 async with session.get('https://additional.url') as additional_response:
-                    additional_data = await r.text()
+                    additional_data = await additional_response.text()
             # ... use response and additional_data to yield items and requests
 
 .. note:: Many libraries that use coroutines, such as `aio-libs`_, require the
diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst
index fc114a63f..d09693c16 100644
--- a/docs/topics/spider-middleware.rst
+++ b/docs/topics/spider-middleware.rst
@@ -98,20 +98,25 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
 
     .. method:: process_spider_output(response, result, spider)
 
+        .. versionchanged:: VERSION
+           Since VERSION this can take and return an :term:`python:asynchronous
+           iterable`.
+
         This method is called with the results returned from the Spider, after
         it has processed the response.
 
-        :meth:`process_spider_output` must return an iterable of
-        :class:`~scrapy.http.Request` objects and :ref:`item object
-        <topics-items>`.
+        :meth:`process_spider_output` must return an iterable (normal or
+        asynchronous) of :class:`~scrapy.http.Request` objects and 
+        :ref:`item objects <topics-items>`.
 
         :param response: the response which generated this output from the
           spider
         :type response: :class:`~scrapy.http.Response` object
 
         :param result: the result returned by the spider
-        :type result: an iterable of :class:`~scrapy.http.Request` objects and
-          :ref:`item object <topics-items>`
+        :type result: an iterable (normal or asynchronous) of
+          :class:`~scrapy.http.Request` objects and :ref:`item objects
+          <topics-items>`
 
         :param spider: the spider whose result is being processed
         :type spider: :class:`~scrapy.spiders.Spider` object
@@ -122,7 +127,7 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
         method (from a previous spider middleware) raises an exception.
 
         :meth:`process_spider_exception` should return either ``None`` or an
-        iterable of :class:`~scrapy.http.Request` objects and :ref:`item object
+        iterable of :class:`~scrapy.http.Request` objects and :ref:`item objects
         <topics-items>`.
 
         If it returns ``None``, Scrapy will continue processing this exception,