Merge pull request #2202 from scrapy/doc-arch-overview2

[MRG+1] DOC move Data Flow below the picture; add links to components

docs/topics/architecture.rst

@@ -16,20 +16,71 @@ components and an outline of the data flow that takes place inside the system
 below with links for more detailed information about them. The data flow is
 also described below.
 
+.. _data-flow:
+
+Data flow
+=========
+
 .. image:: _images/scrapy_architecture_02.png
    :width: 700
    :height: 470
    :alt: Scrapy architecture
 
+The data flow in Scrapy is controlled by the execution engine, and goes like
+this:
+
+1. The :ref:`Engine <component-engine>` gets the first URLs to crawl from the
+   :ref:`Spider <component-spiders>`.
+
+2. The :ref:`Engine <component-engine>` schedules the URLs in the
+   :ref:`Scheduler <component-scheduler>` as Requests and asks for the
+   next URLs to crawl.
+
+3. The :ref:`Scheduler <component-scheduler>` returns the next URLs to crawl
+   to the :ref:`Engine <component-engine>`.
+
+4. The :ref:`Engine <component-engine>` sends the URLs to the
+   :ref:`Downloader <component-downloader>`, passing through the
+   :ref:`Downloader Middleware <component-downloader-middleware>`
+   (request direction).
+
+5. Once the page finishes downloading the
+   :ref:`Downloader <component-downloader>` generates a Response (with
+   that page) and sends it to the Engine, passing through the
+   :ref:`Downloader Middleware <component-downloader-middleware>`
+   (response direction).
+
+6. The :ref:`Engine <component-engine>` receives the Response from the
+   :ref:`Downloader <component-downloader>` and sends it to the
+   :ref:`Spider <component-spiders>` for processing, passing
+   through the :ref:`Spider Middleware <component-spider-middleware>`
+   (input direction).
+
+7. The :ref:`Spider <component-spiders>` processes the Response and returns
+   scraped items and new Requests (to follow) to the
+   :ref:`Engine <component-engine>`, passing through the
+   :ref:`Spider Middleware <component-spider-middleware>` (output direction).
+
+8. The :ref:`Engine <component-engine>` sends processed items to
+   :ref:`Item Pipelines <component-pipelines>` and processed Requests to
+   the :ref:`Scheduler <component-scheduler>`.
+
+9. The process repeats (from step 1) until there are no more requests from the
+   :ref:`Scheduler <component-scheduler>`.
+
 Components
 ==========
 
+.. _component-engine:
+
 Scrapy Engine
 -------------
 
 The engine is responsible for controlling the data flow between all components
-of the system, and triggering events when certain actions occur. See the Data
-Flow section below for more details.
+of the system, and triggering events when certain actions occur. See the
+:ref:`Data Flow <data-flow>` section above for more details.
+
+.. _component-scheduler:
 
 Scheduler
 ---------
@@ -37,12 +88,16 @@ Scheduler
 The Scheduler receives requests from the engine and enqueues them for feeding
 them later (also to the engine) when the engine requests them.
 
+.. _component-downloader:
+
 Downloader
 ----------
 
 The Downloader is responsible for fetching web pages and feeding them to the
 engine which, in turn, feeds them to the spiders.
 
+.. _component-spiders:
+
 Spiders
 -------
 
@@ -50,6 +105,8 @@ Spiders are custom classes written by Scrapy users to parse responses and
 extract items (aka scraped items) from them or additional URLs (requests) to
 follow. For more information see :ref:`topics-spiders`.
 
+.. _component-pipelines:
+
 Item Pipeline
 -------------
 
@@ -58,6 +115,8 @@ extracted (or scraped) by the spiders. Typical tasks include cleansing,
 validation and persistence (like storing the item in a database). For more
 information see :ref:`topics-item-pipeline`.
 
+.. _component-downloader-middleware:
+
 Downloader middlewares
 ----------------------
 
@@ -76,6 +135,8 @@ Use a Downloader middleware if you need to do one of the following:
 
 For more information see :ref:`topics-downloader-middleware`.
 
+.. _component-spider-middleware:
+
 Spider middlewares
 ------------------
 
@@ -93,39 +154,6 @@ Use a Spider middleware if you need to
 
 For more information see :ref:`topics-spider-middleware`.
 
-Data flow
-=========
-
-The data flow in Scrapy is controlled by the execution engine, and goes like
-this:
-
-1. The Engine gets the first URLs to crawl from the Spider.
-
-2. The Engine schedules the URLs in the Scheduler as Requests and asks for the
-   next URLs to crawl.
-
-3. The Scheduler returns the next URLs to crawl to the Engine.
-
-4. The Engine sends the URLs to the Downloader, passing through the
-   Downloader Middleware (request direction).
-
-5. Once the page finishes downloading the Downloader generates a Response (with
-   that page) and sends it to the Engine, passing through the Downloader
-   Middleware (response direction).
-
-6. The Engine receives the Response from the Downloader and sends it to the
-   Spider for processing, passing through the Spider Middleware (input direction).
-
-7. The Spider processes the Response and returns scraped items and new Requests
-   (to follow) to the Engine, passing through the Spider Middleware
-   (output direction).
-
-8. The Engine sends processed items to Item Pipelines and processed Requests to
-   the Scheduler.
-
-9. The process repeats (from step 1) until there are no more requests from the
-   Scheduler.
-
 Event-driven networking
 =======================