Documentation improvements.

This commit is contained in:
Andrey Rakhmatullin 2020-04-28 20:48:20 +05:00
parent 908682497d
commit b5e9122d91
2 changed files with 15 additions and 13 deletions

View File

@ -48,7 +48,7 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):
- :ref:`Signal handlers that support deferreds <signal-deferred>`. - :ref:`Signal handlers that support deferreds <signal-deferred>`.
- The :meth:`start_requests` spider method. - The :meth:`~scrapy.spiders.Spider.start_requests` spider method.
.. _asynchronous generators were introduced in Python 3.6: https://www.python.org/dev/peps/pep-0525/ .. _asynchronous generators were introduced in Python 3.6: https://www.python.org/dev/peps/pep-0525/
@ -59,17 +59,20 @@ Asynchronous start_requests and spider middlewares
.. versionadded:: 2.1 .. versionadded:: 2.1
The :meth:`start_requests` spider method can be an asynchronous generator. In The :meth:`~scrapy.spiders.Spider.start_requests` spider method can be an
this case all spider middlewares used with this spider that have the asynchronous generator. In this case all spider middlewares used with this
spider that have the
:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start_requests` :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start_requests`
method must support this: if they receive an asynchronous iterable, they must method must support this: if they receive an asynchronous iterable, they must
return one as well. On the other hand, if they receive a normal iterable, they return one as well. On the other hand, if they receive a normal iterable, they
shouldn't break. Such universal :meth:`process_start_requests` must be an shouldn't break. Such universal :meth:`process_start_requests` must be an
asynchronous generator itself, and so it will always convert a normal iterable asynchronous generator itself, and so it will always convert a normal iterable
to an asynchronous one. As a result of a middleware method is passed to the to an asynchronous one. Because a result of a middleware method is passed to
same method of the next middleware, it's only possible to mix middlewares with the same method of the next middleware, it's only possible to mix middlewares
synchronous and asynchronous :meth:`process_start_requests` if all synchronous with synchronous and asynchronous :meth:`process_start_requests` if all
ones are called first. synchronous ones are called first.
.. autofunction:: scrapy.utils.asyncgen.as_async_generator
Here is an example of a universal middleware:: Here is an example of a universal middleware::
@ -82,7 +85,7 @@ Here is an example of a universal middleware::
yield r yield r
If this method includes asynchronous code, that code will work even with If this method includes asynchronous code, that code will work even with
synchronous :meth:`start_requests`. synchronous :meth:`~scrapy.spiders.Spider.start_requests`.
Usage Usage
===== =====
@ -123,7 +126,7 @@ This means you can use many useful Python libraries providing such code::
async def parse_with_asyncio(self, response): async def parse_with_asyncio(self, response):
async with aiohttp.ClientSession() as session: async with aiohttp.ClientSession() as session:
async with session.get('https://additional.url') as additional_response: async with session.get('https://additional.url') as additional_response:
additional_data = await r.text() additional_data = await additional_response.text()
# ... use response and additional_data to yield items and requests # ... use response and additional_data to yield items and requests
.. note:: Many libraries that use coroutines, such as `aio-libs`_, require the .. note:: Many libraries that use coroutines, such as `aio-libs`_, require the
@ -133,7 +136,7 @@ This means you can use many useful Python libraries providing such code::
Common use cases for asynchronous code include: Common use cases for asynchronous code include:
* requesting data from websites, databases and other services (in * requesting data from websites, databases and other services (in
:meth:`start_requests`, callbacks, pipelines and middlewares); :meth:`~scrapy.spiders.Spider.start_requests`, callbacks, pipelines and middlewares);
* storing data in databases (in pipelines and middlewares); * storing data in databases (in pipelines and middlewares);
* delaying the spider initialization until some external event (in the * delaying the spider initialization until some external event (in the
:signal:`spider_opened` handler); :signal:`spider_opened` handler);

View File

@ -160,14 +160,13 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
It receives an iterable (in the ``start_requests`` parameter) and must It receives an iterable (in the ``start_requests`` parameter) and must
return another iterable of :class:`~scrapy.http.Request` objects. return another iterable of :class:`~scrapy.http.Request` objects.
When using Python 3.6+ `start_requests` may be an When using Python 3.6+ ``start_requests`` may be an
:term:`python:asynchronous iterable`. To support such spiders, this :term:`python:asynchronous iterable`. To support such spiders, this
middleware method should be an :term:`python:asynchronous generator`, middleware method should be an :term:`python:asynchronous generator`,
which should support both synchronous and asynchronous which should support both synchronous and asynchronous
``start_requests``. :func:`scrapy.utils.asyncgen.as_async_generator` ``start_requests``. :func:`scrapy.utils.asyncgen.as_async_generator`
can be used to convert both kinds of ``start_requests`` to an can be used to convert both kinds of ``start_requests`` to an
asynchronous iterable. See :ref:`more details here asynchronous iterable.
<async-start_requests>`.
.. note:: When implementing this method in your spider middleware, you .. note:: When implementing this method in your spider middleware, you
should always return an iterable (that follows the input one) and should always return an iterable (that follows the input one) and