Documentation improvements.

This commit is contained in:
Andrey Rakhmatullin 2020-04-28 20:48:20 +05:00
parent 908682497d
commit b5e9122d91
2 changed files with 15 additions and 13 deletions

View File

@ -48,7 +48,7 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):
- :ref:`Signal handlers that support deferreds <signal-deferred>`.
- The :meth:`start_requests` spider method.
- The :meth:`~scrapy.spiders.Spider.start_requests` spider method.
.. _asynchronous generators were introduced in Python 3.6: https://www.python.org/dev/peps/pep-0525/
@ -59,17 +59,20 @@ Asynchronous start_requests and spider middlewares
.. versionadded:: 2.1
The :meth:`start_requests` spider method can be an asynchronous generator. In
this case all spider middlewares used with this spider that have the
The :meth:`~scrapy.spiders.Spider.start_requests` spider method can be an
asynchronous generator. In this case all spider middlewares used with this
spider that have the
:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start_requests`
method must support this: if they receive an asynchronous iterable, they must
return one as well. On the other hand, if they receive a normal iterable, they
shouldn't break. Such universal :meth:`process_start_requests` must be an
asynchronous generator itself, and so it will always convert a normal iterable
to an asynchronous one. As a result of a middleware method is passed to the
same method of the next middleware, it's only possible to mix middlewares with
synchronous and asynchronous :meth:`process_start_requests` if all synchronous
ones are called first.
to an asynchronous one. Because a result of a middleware method is passed to
the same method of the next middleware, it's only possible to mix middlewares
with synchronous and asynchronous :meth:`process_start_requests` if all
synchronous ones are called first.
.. autofunction:: scrapy.utils.asyncgen.as_async_generator
Here is an example of a universal middleware::
@ -82,7 +85,7 @@ Here is an example of a universal middleware::
yield r
If this method includes asynchronous code, that code will work even with
synchronous :meth:`start_requests`.
synchronous :meth:`~scrapy.spiders.Spider.start_requests`.
Usage
=====
@ -123,7 +126,7 @@ This means you can use many useful Python libraries providing such code::
async def parse_with_asyncio(self, response):
async with aiohttp.ClientSession() as session:
async with session.get('https://additional.url') as additional_response:
additional_data = await r.text()
additional_data = await additional_response.text()
# ... use response and additional_data to yield items and requests
.. note:: Many libraries that use coroutines, such as `aio-libs`_, require the
@ -133,7 +136,7 @@ This means you can use many useful Python libraries providing such code::
Common use cases for asynchronous code include:
* requesting data from websites, databases and other services (in
:meth:`start_requests`, callbacks, pipelines and middlewares);
:meth:`~scrapy.spiders.Spider.start_requests`, callbacks, pipelines and middlewares);
* storing data in databases (in pipelines and middlewares);
* delaying the spider initialization until some external event (in the
:signal:`spider_opened` handler);

View File

@ -160,14 +160,13 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
It receives an iterable (in the ``start_requests`` parameter) and must
return another iterable of :class:`~scrapy.http.Request` objects.
When using Python 3.6+ `start_requests` may be an
When using Python 3.6+ ``start_requests`` may be an
:term:`python:asynchronous iterable`. To support such spiders, this
middleware method should be an :term:`python:asynchronous generator`,
which should support both synchronous and asynchronous
``start_requests``. :func:`scrapy.utils.asyncgen.as_async_generator`
can be used to convert both kinds of ``start_requests`` to an
asynchronous iterable. See :ref:`more details here
<async-start_requests>`.
asynchronous iterable.
.. note:: When implementing this method in your spider middleware, you
should always return an iterable (that follows the input one) and