diff --git a/docs/_templates/index.html b/docs/_templates/index.html new file mode 100644 index 000000000..e6840aa9d --- /dev/null +++ b/docs/_templates/index.html @@ -0,0 +1,39 @@ +{% extends "defindex.html" %} +{% block tables %} +

Parts of the documentation:

+ + + +
+

Getting started
+ for an overview and tutorial

+

API Reference
+ all details about Scrapy stable API

+

Experimental Features
+ use with caution

+ 		+

Using Scrapy
+ usage guide and key concepts

+

Frequently Asked Questions
+ for common questions

+

Proposed documentation
+ documentation in progress or unstable features

+
+ +

Indices and tables:

+ + +
+

Global Module Index
+ quick access to all modules

+

General Index
+ all functions, classes, terms

+

Glossary
+ the most important terms explained

+ 		+

Search page
+ search this documentation

+

Complete Table of Contents
+ lists all sections and subsections

+
+{% endblock %} diff --git a/docs/_templates/layout.html b/docs/_templates/layout.html new file mode 100644 index 000000000..4d731a7c2 --- /dev/null +++ b/docs/_templates/layout.html @@ -0,0 +1,16 @@ +{% extends "!layout.html" %} + +{% block rootrellink %} + Scrapy v. {{ version }} documentation » +
  • Home |
    • +
  • Getting Started |
    • +
  • Using Scrapy |
    • +
  • API reference |
    • +
  • FAQ |
    • +
  • Search
    • + +{% endblock %} + + + + diff --git a/docs/conf.py b/docs/conf.py index a7e14d28e..f758dbe2e 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -11,7 +11,7 @@ # All configuration values have a default; values that are commented out # serve to show the default. -import sys, os +import sys from os import path # If your extensions are in another directory, add it here. If the directory @@ -29,7 +29,7 @@ sys.path.append(path.join(path.dirname(path.dirname(__file__)), "scrapy")) extensions = ['scrapydocs', 'sphinx.ext.autodoc'] # Add any paths that contain templates here, relative to this directory. -templates_path = ['../templates/docs'] +templates_path = ['_templates'] # The suffix of source filenames. source_suffix = '.rst' @@ -38,7 +38,7 @@ source_suffix = '.rst' #source_encoding = 'utf-8' # The master toctree document. -master_doc = 'index' +master_doc = 'contents' # General information about the project. project = u'Scrapy' @@ -130,7 +130,9 @@ html_use_smartypants = True # Additional templates that should be rendered to pages, maps page names to # template names. -#html_additional_pages = {} +html_additional_pages = { + 'index': 'index.html' +} # If false, no module index is generated. #html_use_modindex = True diff --git a/docs/contents.rst b/docs/contents.rst new file mode 100644 index 000000000..73a97b801 --- /dev/null +++ b/docs/contents.rst @@ -0,0 +1,15 @@ +.. _contents: + +============================= +Scrapy documentation contents +============================= + +.. toctree:: + + intro/index + topics/index + ref/index + faq + topics/adaptors + experimental + proposed/index diff --git a/docs/experimental.rst b/docs/experimental.rst new file mode 100644 index 000000000..1bc4ebea8 --- /dev/null +++ b/docs/experimental.rst @@ -0,0 +1,15 @@ +.. _ref-experimental: + +Experimental features +===================== + +This section documents experimental features that may become stable in future +Scrapy releases, but whose API is not yet stable. Use them with caution, and +subscribe to the `mailing lists `_ to be notified +of any changes. + +.. toctree:: + :maxdepth: 1 + + topics/adaptors + diff --git a/docs/faq.rst b/docs/faq.rst index 45b8976df..cce5a811c 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -29,8 +29,8 @@ Does Scrapy work with Python 3.0? No, and there are no plans to port Scrapy to Python 3.0 yet. At the moment Scrapy works with Python 2.5 or 2.6. -Does Scrapy "stole" X from Django? ----------------------------------- +Did Scrapy "steal" X from Django? +--------------------------------- Probably, but we don't like that word. We think Django_ is a great open source project and an example to follow, so we've used it as an inspiration for diff --git a/docs/index.rst b/docs/index.rst deleted file mode 100644 index 26a0bc9e4..000000000 --- a/docs/index.rst +++ /dev/null @@ -1,75 +0,0 @@ -.. _index: - -============================== -Scrapy |version| documentation -============================== - -Welcome! This is the documentation for Scrapy, a screen scraping framework for -Python. - -For more information about the project visit the `Scrapy homepage -`_. - -Getting started -=============== - -This section helps you get familiarized with Scrapy and its basic concepts. - -.. toctree:: - :maxdepth: 2 - - intro/index - -Using Scrapy -============ - -This section explains all key concepts of Scrapy. - -.. toctree:: - :maxdepth: 2 - - topics/index - -API reference -============= - -This section documents the :ref:`Scrapy API `. See also -:ref:`misc-api-stability`. - -.. toctree:: - :maxdepth: 2 - - ref/index - -Frequently asked questions -========================== - -.. toctree:: - :maxdepth: 2 - - faq - -Experimental features -===================== - -This section documents experimental features that may become stable in future -Scrapy releases, but whose API is not yet stable. Use them with caution, and -subscribe to the `mailing lists `_ to be notified -of any changes. - -.. toctree:: - :maxdepth: 1 - - topics/adaptors - -Proposed documentation -====================== - -This section contains documentation which is not yet completed, or which -overlaps with existing documentation and needs to be manually merged. It may -also contain outdated information, as it's not revised so frequently. - -.. toctree:: - :maxdepth: 2 - - proposed/index diff --git a/docs/intro/overview.rst b/docs/intro/overview.rst index 8391b379f..f5ede250e 100644 --- a/docs/intro/overview.rst +++ b/docs/intro/overview.rst @@ -20,7 +20,7 @@ so you can get an idea of how it works and decide if Scrapy is what you need. When you're ready to start a project, you can :ref:`start with the tutorial `. For more detailed information you can take a look at the -:ref:`documentation contents `. +:ref:`documentation contents `. Pick a website ============== diff --git a/docs/proposed/index.rst b/docs/proposed/index.rst index 2fdf8134b..1f06e8170 100644 --- a/docs/proposed/index.rst +++ b/docs/proposed/index.rst @@ -3,6 +3,10 @@ Proposed documentation ====================== +This section contains documentation which is not yet completed, or which +overlaps with existing documentation and needs to be manually merged. It may +also contain outdated information, as it's not revised so frequently. + .. warning:: This is experimental or "work in progress" documentation, use at your own @@ -14,7 +18,4 @@ Proposed documentation newitem spiders -.. toctree:: - - introduction diff --git a/docs/proposed/introduction.rst b/docs/proposed/introduction.rst deleted file mode 100644 index 7646c3646..000000000 --- a/docs/proposed/introduction.rst +++ /dev/null @@ -1,49 +0,0 @@ -============ -Introduction -============ - -.. architecture: - -Overview -======== - -.. image:: _images/scrapy_architecture.png - :width: 700 - :height: 468 - :alt: Scrapy architecture - -Requests and Responses ----------------------- - -Scrapy uses *Requests* and *Responses* for crawling web sites. - -Generally, *Requests* are generated in the Spiders and pass across the system -until they reach the *Downloader*, which executes the *Request* and returns a -*Response* which goes back to the Spider that generated the *Request*. - -Spiders -------- - -Spiders are user written classes to scrape information from a domain (or group -of domains). - -They define an initial set of URLs (or Requests) to download, how to crawl the -domain and how to scrape *Items* from their pages. - -Items ------ - -Items are the placeholder to use for the scraped data. They are represented by a -simple Python class. - -After an Item has been scraped by a Spider, it is sent to the Item Pipeline for further proccesing. - -Item Pipeline -------------- - -The Item Pipeline is a list of user written Python classes that implement a -specific method, which is called sequentially for every element of the Pipeline. - -Each element receives the Scraped Item, do an action upon it (like validating, -checking for duplicates, store the item), and then decide if the Item continues -trough the Pipeline or the item is dropped. diff --git a/docs/ref/index.rst b/docs/ref/index.rst index 62b7ac591..87d159efd 100644 --- a/docs/ref/index.rst +++ b/docs/ref/index.rst @@ -6,16 +6,17 @@ API Reference This section documents the Scrapy |version| API. For more information see :ref:`misc-api-stability`. .. toctree:: + :maxdepth: 1 + request-response spiders selectors + settings + signals exceptions - request-response + logging + email extension-manager extensions downloader-middleware - settings - signals - logging - email link-extractors