mirror of https://github.com/scrapy/scrapy.git
Merge 8263f5c705 into e28e56aa61
This commit is contained in:
commit
31abc9b812
|
|
@ -203,6 +203,32 @@ This means you can use many useful Python libraries providing such code:
|
|||
.. note:: If you want to ``await`` on Deferreds while using the asyncio reactor,
|
||||
you need to :ref:`wrap them<asyncio-await-dfd>`.
|
||||
|
||||
Sometimes you need to call code that is not async-friendly at all, such as a
|
||||
third-party client library with no asynchronous support. Calling such code
|
||||
directly from a coroutine would block the underlying event loop, preventing
|
||||
Scrapy from doing any other work while it runs. To avoid this, you can wrap
|
||||
the blocking call with :func:`~scrapy.utils.decorators.inthread`, which runs
|
||||
it in a separate thread and returns an awaitable with the result:
|
||||
|
||||
.. skip: next
|
||||
.. code-block:: python
|
||||
|
||||
from scrapy.utils.decorators import inthread
|
||||
|
||||
|
||||
@inthread
|
||||
def blocking_call(value):
|
||||
# e.g. a call into a library with no async support
|
||||
return some_sync_client.get(value)
|
||||
|
||||
|
||||
class MySpider(Spider):
|
||||
async def parse(self, response):
|
||||
result = await blocking_call(response.url)
|
||||
# ... use result to yield items and requests
|
||||
|
||||
.. autofunction:: scrapy.utils.decorators.inthread
|
||||
|
||||
Common use cases for asynchronous code include:
|
||||
|
||||
* requesting data from websites, databases and other services (in
|
||||
|
|
|
|||
Loading…
Reference in New Issue