RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #1 from Gallaecio/asyncio-parse-asyncgen-proper-rebased 

Refactor the asynchronous process_spider_output documentation
This commit is contained in:
Andrey Rahmatullin 2022-07-22 18:28:56 +05:00 committed by GitHub
parent cb3afcf0e7 b95c634b86
commit dc67100a8f
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
7 changed files with 282 additions and 96 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

13
docs/index.rst
View File

@ -229,10 +229,11 @@ Extending Scrapy
topics/downloader-middleware
topics/spider-middleware
topics/extensions
topics/api
topics/signals
topics/scheduler
topics/exporters
topics/components
topics/api
:doc:`topics/architecture`
@ -247,9 +248,6 @@ Extending Scrapy
:doc:`topics/extensions`
Extend Scrapy with your custom functionality
:doc:`topics/api`
Use it on extensions and middlewares to extend Scrapy functionality
:doc:`topics/signals`
See all available signals and how to work with them.
@ -259,6 +257,13 @@ Extending Scrapy
:doc:`topics/exporters`
Quickly export your scraped items to a file (XML, CSV, etc).
:doc:`topics/components`
Learn the common API and some good practices when building custom Scrapy
components.
:doc:`topics/api`
Use it on extensions and middlewares to extend Scrapy functionality
All the rest
============

24
docs/topics/asyncio.rst
View File

@ -96,3 +96,27 @@ Futures. Scrapy provides two helpers for this:
down to Scrapy 2.0 (earlier versions do not support
:mod:`asyncio`), you can copy the implementation of these functions
into your own code.
.. _enforce-asyncio-requirement:
Enforcing asyncio as a requirement
==================================
If you are writing a :ref:`component <topics-components>` that requires asyncio
to work, use :func:`scrapy.utils.reactor.is_asyncio_reactor_installed` to
:ref:`enforce it as a requirement <enforce-component-requirements>`. For
example::
from scrapy.utils.reactor import is_asyncio_reactor_installed
class MyComponent:
def __init__(self):
if not is_asyncio_reactor_installed():
raise ValueError(
f"{MyComponent.__qualname__} requires the asyncio Twisted "
f"reactor. Make sure you have it configured in the "
f"TWISTED_REACTOR setting. See the asyncio documentation "
f"of Scrapy for more information."
)

84
docs/topics/components.rst Normal file
View File

@ -0,0 +1,84 @@
.. _topics-components:
==========
Components
==========
A Scrapy component is any class whose objects are created using
:func:`scrapy.utils.misc.create_instance`.
That includes the classes that you may assign to the following settings:
- :setting:`DNS_RESOLVER`
- :setting:`DOWNLOAD_HANDLERS`
- :setting:`DOWNLOADER_CLIENTCONTEXTFACTORY`
- :setting:`DOWNLOADER_MIDDLEWARES`
- :setting:`DUPEFILTER_CLASS`
- :setting:`EXTENSIONS`
- :setting:`FEED_EXPORTERS`
- :setting:`FEED_STORAGES`
- :setting:`ITEM_PIPELINES`
- :setting:`SCHEDULER`
- :setting:`SCHEDULER_DISK_QUEUE`
- :setting:`SCHEDULER_MEMORY_QUEUE`
- :setting:`SCHEDULER_PRIORITY_QUEUE`
- :setting:`SPIDER_MIDDLEWARES`
Third-party Scrapy components may also let you define additional Scrapy
components, usually configurable through :ref:`settings <topics-settings>`, to
modify their behavior.
.. _enforce-component-requirements:
Enforcing component requirements
================================
Sometimes, your components may only be intended to work under certain
conditions. For example, the may require a minimum version of Scrapy to work as
intended, or they may require certain settings to have specific values.
In addition to describing those conditions in the documentation of your
component, it is a good practice to raise an exception from the ``__init__``
method of your component if those conditions are not met at run time.
In the case of :ref:`downloader middlewares <topics-downloader-middleware>`,
:ref:`extensions <topics-extensions>`, :ref:`item pipelines
<topics-item-pipeline>`, and :ref:`spider middlewares
<topics-spider-middleware>`, you should raise
:exc:`scrapy.exceptions.NotConfigured`, passing a description of the issue as a
parameter to the exception so that it is printed in the logs, for the user to
see. For other components, feel free to raise whatever other exception feels
right to you; for example, :exc:`RuntimeError` would make sense for a Scrapy
version mismatch, while :exc:`ValueError` may be better if the issue is the
value of a setting.
If your requirement is a minimum Scrapy version, you may use
:attr:`scrapy.__version__` to enforce your requirement. For example::
from pkg_resources import parse_version
import scrapy
class MyComponent:
def __init__(self):
if parse_version(scrapy.__version__) < parse_version('VERSION'):
raise RuntimeError(
f"{MyComponent.__qualname__} requires Scrapy VERSION or "
f"later, which allow defining the process_spider_output "
f"method of spider middlewares as an asynchronous "
f"generator."
)

160
docs/topics/coroutines.rst
View File

@ -17,9 +17,12 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):
- :class:`~scrapy.Request` callbacks.
If you are using any custom or third-party :ref:`spider middleware
<topics-spider-middleware>`, see :ref:`sync-async-spider-middleware`.
.. versionchanged:: VERSION
Output of async callbacks is now processed asynchronously instead of collecting
all of it first.
Output of async callbacks is now processed asynchronously instead of
collecting all of it first.
- The :meth:`process_item` method of
:ref:`item pipelines <topics-item-pipeline>`.
@ -34,18 +37,26 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):
- :ref:`Signal handlers that support deferreds <signal-deferred>`.
- The :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output`
method of :ref:`spider middlewares <custom-spider-middleware>`. See
:ref:`async-spider-middlewares`.
- The
:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output`
method of :ref:`spider middlewares <topics-spider-middleware>`.
It must be defined as an :term:`asynchronous generator`. The input
``result`` parameter is an :term:`asynchronous iterable`.
See also :ref:`sync-async-spider-middleware` and
:ref:`universal-spider-middleware`.
.. versionadded:: VERSION
Usage
=====
General usage
=============
There are several use cases for coroutines in Scrapy. Code that would
return Deferreds when written for previous Scrapy versions, such as downloader
middlewares and signal handlers, can be rewritten to be shorter and cleaner::
There are several use cases for coroutines in Scrapy.
Code that would return Deferreds when written for previous Scrapy versions,
such as downloader middlewares and signal handlers, can be rewritten to be
shorter and cleaner::
from itemadapter import ItemAdapter
@ -110,46 +121,76 @@ Common use cases for asynchronous code include:
.. _aio-libs: https://github.com/aio-libs
.. _async-spider-middlewares:
Asynchronous spider middlewares
===============================
.. _sync-async-spider-middleware:
Mixing synchronous and asynchronous spider middlewares
======================================================
.. versionadded:: VERSION
.. note:: This currently applies to
:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output`.
In the future it will also apply to
:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start_requests`.
Middleware methods discussed here can take and return async iterables. They can
return the same type of iterable or they can take a normal one and return an
async one. If such method needs to return an async iterable it must be an async
generator, not just a coroutine that returns an iterable.
The output of a :class:`~scrapy.Request` callback is passed as the ``result``
parameter to the
:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output` method
of the first :ref:`spider middleware <topics-spider-middleware>` from the
:ref:`list of active spider middlewares <topics-spider-middleware-setting>`.
Then the output of that ``process_spider_output`` method is passed to the
``process_spider_output`` method of the next spider middleware, and so on for
every active spider middleware.
As the result of a middleware method is passed to the same method of the next
middleware, it needs to be adapted if the second method expects a different
type. Scrapy will do this transparently:
Scrapy supports mixing :ref:`coroutine methods <async>` and synchronous methods
in this chain of calls.
* A normal iterable is wrapped into an async one which shouldn't cause any side
effects.
* An async iterable is downgraded to a normal one by waiting until all results
are available and wrapping them in a normal iterable. This is problematic
because it pauses the normal middleware processing for this iterable and
because all results can be skipped if exceptions are raised during
processing. This case emits a warning and will be deprecated and then removed
in a later Scrapy version.
* Async iterables returned from
:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_exception`
won't be downgraded, an exception will be raised if that is needed.
However, if any of the ``process_spider_output`` methods is defined as a
synchronous method, and the previous ``Request`` callback or
``process_spider_output`` method is a coroutine, there are some drawbacks to
the asynchronous-to-synchronous conversion that Scrapy does so that the
synchronous ``process_spider_output`` method gets a synchronous iterable as its
``result`` parameter:
As downgrading is undesirable, here is the proposed way to avoid it. If all
middlewares, including 3rd-party ones, support async iterables as input, no
downgrading will happen. But removing normal iterable support (making the
method a coroutine) from a middleware published as a separate project or used
internally in projects for older Scrapy versions breaks backwards
compatibility. So, as an interim measure (it will be deprecated and then
removed in a later Scrapy version), a middleware can provide both sync and
async methods in the following form::
- The whole output of the previous ``Request`` callback or
``process_spider_output`` method is awaited at this point.
- If an exception raises while awaiting the output of the previous
``Request`` callback or ``process_spider_output`` method, none of that
output will be processed.
This contrasts with the regular behavior, where all items yielded before
an exception raises are processed.
Asynchronous-to-synchronous conversions are supported for backward
compatibility, but they are deprecated and will stop working in a future
version of Scrapy.
To avoid asynchronous-to-synchronous conversions, when defining ``Request``
callbacks as coroutine methods or when using spider middlewares whose
``process_spider_output`` method is an :term:`asynchronous generator`, all
active spider middlewares must either have their ``process_spider_output``
method defined as an asynchronous generator or :ref:`define a
process_spider_output_async method <universal-spider-middleware>`.
.. note:: When using third-party spider middlewares that only define a
synchronous ``process_spider_output`` method, consider
:ref:`making them universal <universal-spider-middleware>` through
:ref:`subclassing <tut-inheritance>`.
.. _universal-spider-middleware:
Universal spider middlewares
============================
.. versionadded:: VERSION
To allow writing a spider middleware that supports asynchronous execution of
its ``process_spider_output`` method in Scrapy VERSION and later (avoiding
:ref:`asynchronous-to-synchronous conversions <sync-async-spider-middleware>`)
while maintaining support for older Scrapy versions, you may define
``process_spider_output`` as a synchronous method and define an
:term:`asynchronous generator` version of that method with an alternative name:
``process_spider_output_async``.
For example::
class UniversalSpiderMiddleware:
def process_spider_output(self, response, result, spider):
@ -162,28 +203,13 @@ async methods in the following form::
# ... do something with r
yield r
In this case normal and async iterables will be passed to the respective
methods without any wrapping or downgrading, and in older versions of Scrapy
the coroutine method will just be ignored. When the backwards compatibility is
no longer needed the non-coroutine method can be dropped and the coroutine one
renamed to the normal name. It may be possible to extract common code from both
methods to reduce code duplication, as in the simplest case the only difference
between them will be ``for`` vs ``async for``.
.. note:: This is an interim measure to allow, for a time, to write code that
works in Scrapy VERSION and later without requiring
asynchronous-to-synchronous conversions, and works in earlier Scrapy
versions as well.
So, to recap:
* If you don't intend to use async callbacks or middlewares containing async
code in your project, nothing should change for you yet. At some point in the
future some of the 3rd-party middlewares you use may drop backwards
compatibility, which shouldn't lead to immediate problems but may be a sign
to start converting your code to ``async def`` too.
* If you maintain a middleware that can be used with projects you can't control
(e.g. one you published for other people to use, or one that needs to support
some old project that can't be modernized), we recommend adding a
``process_spider_output_async`` method so that the amount of unnecessary
iterable conversions is reduced but no compatibility is broken.
* If you use async callbacks, try to make sure all middlewares support them.
Note that you can modernize 3rd-party middlewares by subclassing them.
* If you want to write and publish a middleware that requires async code, you
should write in the docs that the minimum support Scrapy version is VERSION
(maybe even check this at the run time, using :attr:`scrapy.__version__`).
In some future version of Scrapy, however, this feature will be
deprecated and, eventually, in a later version of Scrapy, this
feature will be removed, and all spider middlewares will be expected
to define their ``process_spider_output`` method as an asynchronous
generator.

24
docs/topics/spider-middleware.rst
View File

@ -98,10 +98,6 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
.. method:: process_spider_output(response, result, spider)
.. versionchanged:: VERSION
Since VERSION this can take and return an :term:`python:asynchronous
iterable`.
This method is called with the results returned from the Spider, after
it has processed the response.
@ -109,8 +105,17 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
:class:`~scrapy.Request` objects and :ref:`item objects
<topics-items>`.
.. note:: When defined as a :ref:`coroutine <async>`, this method needs
to be an async generator, not just return an iterable.
.. versionchanged:: VERSION
This method may be defined as an :term:`asynchronous generator`, in
which case ``result`` is an :term:`asynchronous iterable`.
Consider defining this method as an :term:`asynchronous generator`,
which will be a requirement in a future version of Scrapy. However, if
you plan on sharing your spider middleware with other people, consider
either :ref:`enforcing Scrapy VERSION <enforce-component-requirements>`
as a minimum requirement of your spider middleware, or :ref:`making
your spider middleware universal <universal-spider-middleware>` so that
it works with Scrapy versions earlier than Scrapy VERSION.
:param response: the response which generated this output from the
spider
@ -127,10 +132,9 @@ object gives you access, for example, to the :ref:`settings <topics-settings>`.
.. versionadded:: VERSION
If exists, this methid will be called instead of
:meth:`process_spider_output` when ``result`` is an async iterable.
If this method exists, it must be a coroutine while
:meth:`process_spider_output` must not be a coroutine.
If defined, this method must be an :term:`asynchronous generator`,
which will be called instead of :meth:`process_spider_output` if
``result`` is an :term:`asynchronous iterable`.
.. method:: process_spider_exception(response, exception, spider)

19
scrapy/core/spidermw.py
View File

@ -4,7 +4,7 @@ Spider Middleware manager
See documentation in docs/topics/spider-middleware.rst
"""
import logging
from inspect import isasyncgenfunction
from inspect import isasyncgenfunction, iscoroutine
from itertools import islice
from typing import Any, AsyncGenerator, AsyncIterable, Callable, Generator, Iterable, Tuple, Union, cast
@ -61,7 +61,7 @@ class SpiderMiddlewareManager(MiddlewareManager):
try:
result = method(response=response, spider=spider)
if result is not None:
msg = (f"Middleware {method.__qualname__} must return None "
msg = (f"{method.__qualname__} must return None "
f"or raise an exception, got {type(result)}")
raise _InvalidOutput(msg)
except _InvalidOutput:
@ -129,7 +129,7 @@ class SpiderMiddlewareManager(MiddlewareManager):
elif result is None:
continue
else:
msg = (f"Middleware {method.__qualname__} must return None "
msg = (f"{method.__qualname__} must return None "
f"or an iterable, got {type(result)}")
raise _InvalidOutput(msg)
return _failure
@ -197,8 +197,17 @@ class SpiderMiddlewareManager(MiddlewareManager):
if _isiterable(result):
result = self._evaluate_iterable(response, spider, result, method_index + 1, recovered)
else:
msg = (f"Middleware {method.__qualname__} must return an "
f"iterable, got {type(result)}")
if iscoroutine(result):
result.close() # Silence warning about not awaiting
msg = (
f"{method.__qualname__} must be an asynchronous "
f"generator (i.e. use yield)"
)
else:
msg = (
f"{method.__qualname__} must return an iterable, got "
f"{type(result)}"
)
raise _InvalidOutput(msg)
last_result_is_async = isinstance(result, AsyncIterable)

54
tests/test_spidermiddleware.py
View File

@ -123,6 +123,11 @@ class BaseAsyncSpiderMiddlewareTestCase(SpiderMiddlewareTestCase):
start_index = 10
return {i: c for c, i in enumerate(mw_classes, start=start_index)}
def _scrape_func(self, *args, **kwargs):
yield {'foo': 1}
yield {'foo': 2}
yield {'foo': 3}
@defer.inlineCallbacks
def _get_middleware_result(self, *mw_classes, start_index: Optional[int] = None):
setting = self._construct_mw_setting(*mw_classes, start_index=start_index)
@ -201,11 +206,6 @@ class ProcessSpiderOutputSimple(BaseAsyncSpiderMiddlewareTestCase):
MW_ASYNCGEN = ProcessSpiderOutputAsyncGenMiddleware
MW_UNIVERSAL = ProcessSpiderOutputUniversalMiddleware
def _scrape_func(self, *args, **kwargs):
yield {'foo': 1}
yield {'foo': 2}
yield {'foo': 3}
def test_simple(self):
""" Simple mw """
return self._test_simple_base(self.MW_SIMPLE)
@ -285,6 +285,45 @@ class ProcessSpiderOutputAsyncGen(ProcessSpiderOutputSimple):
downgrade=True)
class ProcessSpiderOutputNonIterableMiddleware:
def process_spider_output(self, response, result, spider):
return
class ProcessSpiderOutputCoroutineMiddleware:
async def process_spider_output(self, response, result, spider):
results = []
for r in result:
results.append(r)
return results
class ProcessSpiderOutputInvalidResult(BaseAsyncSpiderMiddlewareTestCase):
@defer.inlineCallbacks
def test_non_iterable(self):
with self.assertRaisesRegex(
_InvalidOutput,
(
"\.process_spider_output must return an iterable, got <class "
"'NoneType'>"
),
):
yield self._get_middleware_result(
ProcessSpiderOutputNonIterableMiddleware,
)
@defer.inlineCallbacks
def test_coroutine(self):
with self.assertRaisesRegex(
_InvalidOutput,
"\.process_spider_output must be an asynchronous generator",
):
yield self._get_middleware_result(
ProcessSpiderOutputCoroutineMiddleware,
)
class ProcessStartRequestsSimpleMiddleware:
def process_start_requests(self, start_requests, spider):
for r in start_requests:
@ -387,11 +426,6 @@ class BuiltinMiddlewareSimpleTest(BaseAsyncSpiderMiddlewareTestCase):
MW_ASYNCGEN = ProcessSpiderOutputAsyncGenMiddleware
MW_UNIVERSAL = ProcessSpiderOutputUniversalMiddleware
def _scrape_func(self, *args, **kwargs):
yield {'foo': 1}
yield {'foo': 2}
yield {'foo': 3}
@defer.inlineCallbacks
def _get_middleware_result(self, *mw_classes, start_index: Optional[int] = None):
setting = self._construct_mw_setting(*mw_classes, start_index=start_index)