mirror of https://github.com/scrapy/scrapy.git
Add docs about the new queue behavior.
This commit is contained in:
parent
3141df5628
commit
914726f696
|
|
@ -116,6 +116,31 @@ iterables and choose one at run time::
|
|||
return self._normal_process_start_requests(start_requests, spider)
|
||||
|
||||
|
||||
.. _async-start_requests-queue:
|
||||
|
||||
Queue behavior with asynchronous start_requests
|
||||
===============================================
|
||||
|
||||
.. versionadded:: 2.x
|
||||
|
||||
By default the requests produced by
|
||||
:meth:`~scrapy.spiders.Spider.start_requests` don't follow the usual request
|
||||
priorities: they are only scheduled while the downloader queue is empty. In
|
||||
practice this often means that after some initial requests are scheduled (the
|
||||
number depends on the spider concurrency settings), only the requests produced
|
||||
by callbacks are scheduled. This behavior is inconvenient, confusing and
|
||||
requires workarounds (such as calling ``self.crawler.engine.schedule(request)``
|
||||
directly), but changing it would be backwards-incompatible.
|
||||
|
||||
A better behavior, which should treat requests from
|
||||
:meth:`~scrapy.spiders.Spider.start_requests` in the same way as other
|
||||
requests, is enabled when :meth:`~scrapy.spiders.Spider.start_requests` is an
|
||||
async function (declared using ``async def``). It doesn't need to contain
|
||||
``await`` for this to work, so if you want the new queue behavior, you can just
|
||||
change ``def`` to ``async def``. Note though, that using ``yield`` statements
|
||||
inside an async function makes it an async generator which are only supported
|
||||
since Python 3.6.
|
||||
|
||||
Usage
|
||||
=====
|
||||
|
||||
|
|
|
|||
|
|
@ -169,6 +169,14 @@ scrapy.Spider
|
|||
# each of them, with another callback
|
||||
pass
|
||||
|
||||
.. note::
|
||||
.. versionchanged:: 2.x
|
||||
|
||||
By default the requests produced by :meth:`start_requests` can be
|
||||
processed much later than expected. For :meth:`start_requests`
|
||||
declared as an ``async def`` function a better behavior is used. See
|
||||
:ref:`async-start_requests-queue` for details.
|
||||
|
||||
.. method:: parse(response)
|
||||
|
||||
This is the default callback used by Scrapy to process downloaded
|
||||
|
|
|
|||
Loading…
Reference in New Issue