From 914726f6969509fb63b1eb178276fe93d0054445 Mon Sep 17 00:00:00 2001
From: Andrey Rakhmatullin <wrar@wrar.name>
Date: Fri, 7 Aug 2020 20:40:54 +0500
Subject: [PATCH] Add docs about the new queue behavior.

---
 docs/topics/coroutines.rst | 25 +++++++++++++++++++++++++
 docs/topics/spiders.rst    |  8 ++++++++
 2 files changed, 33 insertions(+)

diff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rst
index 009e21dcf..0cd607bd8 100644
--- a/docs/topics/coroutines.rst
+++ b/docs/topics/coroutines.rst
@@ -116,6 +116,31 @@ iterables and choose one at run time::
                 return self._normal_process_start_requests(start_requests, spider)
 
 
+.. _async-start_requests-queue:
+
+Queue behavior with asynchronous start_requests
+===============================================
+
+.. versionadded:: 2.x
+
+By default the requests produced by
+:meth:`~scrapy.spiders.Spider.start_requests` don't follow the usual request
+priorities: they are only scheduled while the downloader queue is empty. In
+practice this often means that after some initial requests are scheduled (the
+number depends on the spider concurrency settings), only the requests produced
+by callbacks are scheduled. This behavior is inconvenient, confusing and
+requires workarounds (such as calling ``self.crawler.engine.schedule(request)``
+directly), but changing it would be backwards-incompatible.
+
+A better behavior, which should treat requests from
+:meth:`~scrapy.spiders.Spider.start_requests` in the same way as other
+requests, is enabled when :meth:`~scrapy.spiders.Spider.start_requests` is an
+async function (declared using ``async def``). It doesn't need to contain
+``await`` for this to work, so if you want the new queue behavior, you can just
+change ``def`` to ``async def``. Note though, that using ``yield`` statements
+inside an async function makes it an async generator which are only supported
+since Python 3.6.
+
 Usage
 =====
 
diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst
index e50e4aa0a..51f596e5c 100644
--- a/docs/topics/spiders.rst
+++ b/docs/topics/spiders.rst
@@ -169,6 +169,14 @@ scrapy.Spider
                    # each of them, with another callback
                    pass
 
+       .. note::
+           .. versionchanged:: 2.x
+
+           By default the requests produced by :meth:`start_requests` can be
+           processed much later than expected. For :meth:`start_requests`
+           declared as an ``async def`` function a better behavior is used. See
+           :ref:`async-start_requests-queue` for details.
+
    .. method:: parse(response)
 
        This is the default callback used by Scrapy to process downloaded