From 8f4a268f3757fe663de49e80dfd9325c260fcc73 Mon Sep 17 00:00:00 2001 From: Elias Dorneles Date: Thu, 26 Mar 2015 12:14:56 -0300 Subject: [PATCH] added bit about async requests, improved phrasing --- docs/intro/overview.rst | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/docs/intro/overview.rst b/docs/intro/overview.rst index 9a3015ddc..295208bd2 100644 --- a/docs/intro/overview.rst +++ b/docs/intro/overview.rst @@ -74,10 +74,16 @@ attribute (in this case, only the URL for StackOverflow top questions page), and then called the default callback method ``parse`` passing the response object as an argument. -In the ``parse`` callback, we scrape the links to the questions and -yield a few more requests to be processed, registering for them -the method ``parse_question`` as the callback to be called when the -requests are complete. +Here you notice one of the main advantages about Scrapy: requests are +scheduled and processed asynchronously. This means that Scrapy doesn't +need to wait for a request to be finished and processed, it can send +another request or do other things in the meantime, which results in much +faster crawlings. + +So, in the ``parse`` callback, we scrape the links to the questions and +yield a few more requests to be done, registering for them the method +``parse_question`` as the callback to be called for each of them as +they finish. Finally, the ``parse_question`` callback scrapes the question data for each page yielding a dict, which Scrapy then collects and