mirror of https://github.com/scrapy/scrapy.git
Documentation improvements.
This commit is contained in:
parent
908682497d
commit
b5e9122d91
|
|
@ -48,7 +48,7 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):
|
||||||
|
|
||||||
- :ref:`Signal handlers that support deferreds <signal-deferred>`.
|
- :ref:`Signal handlers that support deferreds <signal-deferred>`.
|
||||||
|
|
||||||
- The :meth:`start_requests` spider method.
|
- The :meth:`~scrapy.spiders.Spider.start_requests` spider method.
|
||||||
|
|
||||||
.. _asynchronous generators were introduced in Python 3.6: https://www.python.org/dev/peps/pep-0525/
|
.. _asynchronous generators were introduced in Python 3.6: https://www.python.org/dev/peps/pep-0525/
|
||||||
|
|
||||||
|
|
@ -59,17 +59,20 @@ Asynchronous start_requests and spider middlewares
|
||||||
|
|
||||||
.. versionadded:: 2.1
|
.. versionadded:: 2.1
|
||||||
|
|
||||||
The :meth:`start_requests` spider method can be an asynchronous generator. In
|
The :meth:`~scrapy.spiders.Spider.start_requests` spider method can be an
|
||||||
this case all spider middlewares used with this spider that have the
|
asynchronous generator. In this case all spider middlewares used with this
|
||||||
|
spider that have the
|
||||||
:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start_requests`
|
:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start_requests`
|
||||||
method must support this: if they receive an asynchronous iterable, they must
|
method must support this: if they receive an asynchronous iterable, they must
|
||||||
return one as well. On the other hand, if they receive a normal iterable, they
|
return one as well. On the other hand, if they receive a normal iterable, they
|
||||||
shouldn't break. Such universal :meth:`process_start_requests` must be an
|
shouldn't break. Such universal :meth:`process_start_requests` must be an
|
||||||
asynchronous generator itself, and so it will always convert a normal iterable
|
asynchronous generator itself, and so it will always convert a normal iterable
|
||||||
to an asynchronous one. As a result of a middleware method is passed to the
|
to an asynchronous one. Because a result of a middleware method is passed to
|
||||||
same method of the next middleware, it's only possible to mix middlewares with
|
the same method of the next middleware, it's only possible to mix middlewares
|
||||||
synchronous and asynchronous :meth:`process_start_requests` if all synchronous
|
with synchronous and asynchronous :meth:`process_start_requests` if all
|
||||||
ones are called first.
|
synchronous ones are called first.
|
||||||
|
|
||||||
|
.. autofunction:: scrapy.utils.asyncgen.as_async_generator
|
||||||
|
|
||||||
Here is an example of a universal middleware::
|
Here is an example of a universal middleware::
|
||||||
|
|
||||||
|
|
@ -82,7 +85,7 @@ Here is an example of a universal middleware::
|
||||||
yield r
|
yield r
|
||||||
|
|
||||||
If this method includes asynchronous code, that code will work even with
|
If this method includes asynchronous code, that code will work even with
|
||||||
synchronous :meth:`start_requests`.
|
synchronous :meth:`~scrapy.spiders.Spider.start_requests`.
|
||||||
|
|
||||||
Usage
|
Usage
|
||||||
=====
|
=====
|
||||||
|
|
@ -123,7 +126,7 @@ This means you can use many useful Python libraries providing such code::
|
||||||
async def parse_with_asyncio(self, response):
|
async def parse_with_asyncio(self, response):
|
||||||
async with aiohttp.ClientSession() as session:
|
async with aiohttp.ClientSession() as session:
|
||||||
async with session.get('https://additional.url') as additional_response:
|
async with session.get('https://additional.url') as additional_response:
|
||||||
additional_data = await r.text()
|
additional_data = await additional_response.text()
|
||||||
# ... use response and additional_data to yield items and requests
|
# ... use response and additional_data to yield items and requests
|
||||||
|
|
||||||
.. note:: Many libraries that use coroutines, such as `aio-libs`_, require the
|
.. note:: Many libraries that use coroutines, such as `aio-libs`_, require the
|
||||||
|
|
@ -133,7 +136,7 @@ This means you can use many useful Python libraries providing such code::
|
||||||
Common use cases for asynchronous code include:
|
Common use cases for asynchronous code include:
|
||||||
|
|
||||||
* requesting data from websites, databases and other services (in
|
* requesting data from websites, databases and other services (in
|
||||||
:meth:`start_requests`, callbacks, pipelines and middlewares);
|
:meth:`~scrapy.spiders.Spider.start_requests`, callbacks, pipelines and middlewares);
|
||||||
* storing data in databases (in pipelines and middlewares);
|
* storing data in databases (in pipelines and middlewares);
|
||||||
* delaying the spider initialization until some external event (in the
|
* delaying the spider initialization until some external event (in the
|
||||||
:signal:`spider_opened` handler);
|
:signal:`spider_opened` handler);
|
||||||
|
|
|
||||||
|
|
@ -160,14 +160,13 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
|
||||||
It receives an iterable (in the ``start_requests`` parameter) and must
|
It receives an iterable (in the ``start_requests`` parameter) and must
|
||||||
return another iterable of :class:`~scrapy.http.Request` objects.
|
return another iterable of :class:`~scrapy.http.Request` objects.
|
||||||
|
|
||||||
When using Python 3.6+ `start_requests` may be an
|
When using Python 3.6+ ``start_requests`` may be an
|
||||||
:term:`python:asynchronous iterable`. To support such spiders, this
|
:term:`python:asynchronous iterable`. To support such spiders, this
|
||||||
middleware method should be an :term:`python:asynchronous generator`,
|
middleware method should be an :term:`python:asynchronous generator`,
|
||||||
which should support both synchronous and asynchronous
|
which should support both synchronous and asynchronous
|
||||||
``start_requests``. :func:`scrapy.utils.asyncgen.as_async_generator`
|
``start_requests``. :func:`scrapy.utils.asyncgen.as_async_generator`
|
||||||
can be used to convert both kinds of ``start_requests`` to an
|
can be used to convert both kinds of ``start_requests`` to an
|
||||||
asynchronous iterable. See :ref:`more details here
|
asynchronous iterable.
|
||||||
<async-start_requests>`.
|
|
||||||
|
|
||||||
.. note:: When implementing this method in your spider middleware, you
|
.. note:: When implementing this method in your spider middleware, you
|
||||||
should always return an iterable (that follows the input one) and
|
should always return an iterable (that follows the input one) and
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue