RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

add proper example 

Co-authored-by: Adrián Chaves <adrian@chaves.io>
This commit is contained in:
Bulat Khabibullin 2023-05-11 12:53:43 +05:00 committed by GitHub
parent 26374e21f8
commit 85103b4932
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 6 additions and 7 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

13
docs/topics/asyncio.rst
View File

@ -123,6 +123,8 @@ You can also send multiple requests in parallel:
.. code-block:: python
from twisted.internet.defer import DeferredList
class BatchRequestsSpider(scrapy.Spider):
name = "batch"
start_urls = ["https://example.com/product"]
@ -132,14 +134,11 @@ You can also send multiple requests in parallel:
scrapy.Request("https://example.com/price"),
scrapy.Request("https://example.com/color"),
]
coroutines = []
deferreds = []
for r in additional_requests:
deffered = self.crawler.engine.download(r)
coroutines.append(maybe_deferred_to_future(deffered))
responses = await asyncio.gather(
*coroutines, return_exceptions=True
)
deferred = self.crawler.engine.download(r)
deferreds.append(deferred)
responses = await maybe_deferred_to_future(DeferredList(deferreds))
yield {
'h1': response.css('h1::text').get(),
'price': responses[0].css('.price::text').get(),