From 2c2dc6176602d8e78dcad027b7db0714ec68a884 Mon Sep 17 00:00:00 2001 From: Pablo Hoffman Date: Wed, 7 Jan 2009 03:59:39 +0000 Subject: [PATCH] somes fixes and updates to scrapy documentation --HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40669 --- scrapy/trunk/docs/faq.rst | 4 ++-- scrapy/trunk/docs/index.rst | 6 ++---- scrapy/trunk/docs/intro/tutorial/tutorial1.rst | 8 ++++---- scrapy/trunk/docs/intro/tutorial/tutorial2.rst | 4 ++-- scrapy/trunk/docs/intro/tutorial/tutorial3.rst | 4 ++-- scrapy/trunk/docs/intro/tutorial/tutorial4.rst | 2 +- scrapy/trunk/docs/topics/item-pipeline.rst | 9 ++++++--- 7 files changed, 19 insertions(+), 18 deletions(-) diff --git a/scrapy/trunk/docs/faq.rst b/scrapy/trunk/docs/faq.rst index c2b5682a5..ac8247ff6 100644 --- a/scrapy/trunk/docs/faq.rst +++ b/scrapy/trunk/docs/faq.rst @@ -26,6 +26,6 @@ framework). Does Scrapy work with Python 3.0? --------------------------------- -No, and there is no plan to port Scrapy to Python 3.0 yet. At the moment Scrapy -requires Python 2.5 or 2.6. +No, and there are no plans to port Scrapy to Python 3.0 yet. At the moment +Scrapy requires Python 2.5 or 2.6. diff --git a/scrapy/trunk/docs/index.rst b/scrapy/trunk/docs/index.rst index 0424db814..1c4be7204 100644 --- a/scrapy/trunk/docs/index.rst +++ b/scrapy/trunk/docs/index.rst @@ -1,11 +1,9 @@ -.. Scrapy documentation master file, created by sphinx-quickstart on Mon Nov 24 12:02:52 2008. - You can adapt this file completely to your liking, but it should at least - contain the root `toctree` directive. +.. _index: Scrapy |version| documentation ================================ -Welcome! This is the documentation for Scrapy |version|. +Welcome! This is the documentation for Scrapy |version|, last updated on |today|. For more information visit the `Scrapy homepage `_. diff --git a/scrapy/trunk/docs/intro/tutorial/tutorial1.rst b/scrapy/trunk/docs/intro/tutorial/tutorial1.rst index 97534d538..0e4498caf 100644 --- a/scrapy/trunk/docs/intro/tutorial/tutorial1.rst +++ b/scrapy/trunk/docs/intro/tutorial/tutorial1.rst @@ -1,11 +1,11 @@ -.. _tutorial1: - -.. highlight:: sh +.. _intro-tutorial1: ====================== Creating a new project ====================== +.. highlight:: sh + In this tutorial, we'll teach you how to scrape http://www.google.com/dirhp Google's web directory. We'll assume that Scrapy is already installed in your system, if not see :ref:`intro-install`. @@ -29,4 +29,4 @@ As long as Scrapy is well installed and the path is set, this should create a di $ export PYTHONPATH=/path/to/your/project -Now you can continue with the next part of the tutorial: :ref:`tutorial2`. +Now you can continue with the next part of the tutorial: :ref:`intro-tutorial2`. diff --git a/scrapy/trunk/docs/intro/tutorial/tutorial2.rst b/scrapy/trunk/docs/intro/tutorial/tutorial2.rst index e5776f76b..e0ef25925 100644 --- a/scrapy/trunk/docs/intro/tutorial/tutorial2.rst +++ b/scrapy/trunk/docs/intro/tutorial/tutorial2.rst @@ -1,4 +1,4 @@ -.. _tutorial2: +.. _intro-tutorial2: ================ Our first spider @@ -121,5 +121,5 @@ You can try crawling with this little code, by running:: ./scrapy-ctl crawl google.com and it will actually work, altough it won't do any parsing, since parse_category is not defined, and that's exactly what we're going to do in the next part of -the tutorial: :ref:`tutorial3`. +the tutorial: :ref:`intro-tutorial3`. diff --git a/scrapy/trunk/docs/intro/tutorial/tutorial3.rst b/scrapy/trunk/docs/intro/tutorial/tutorial3.rst index de6da1673..f7d80f2df 100644 --- a/scrapy/trunk/docs/intro/tutorial/tutorial3.rst +++ b/scrapy/trunk/docs/intro/tutorial/tutorial3.rst @@ -1,4 +1,4 @@ -.. _tutorial3: +.. _intro-tutorial3: ================= Scraping our data @@ -71,4 +71,4 @@ are handled different than others (in fact, it *will* happen once you scrape mor The rest of the code is quite self-explanatory. The *attribute* method sets the item's attributes, and the items themselves are put into a list that we'll return to Scrapy's engine. One simple (although important) thing to remember here is that you must always return a list that contains either items, requests, or both, but always inside a list. -So, we're almost done! Let's now check the last part of the tutorial: :ref:`tutorial4` +So, we're almost done! Let's now check the last part of the tutorial: :ref:`intro-tutorial4` diff --git a/scrapy/trunk/docs/intro/tutorial/tutorial4.rst b/scrapy/trunk/docs/intro/tutorial/tutorial4.rst index 8387569bc..3aba9bbdc 100644 --- a/scrapy/trunk/docs/intro/tutorial/tutorial4.rst +++ b/scrapy/trunk/docs/intro/tutorial/tutorial4.rst @@ -1,4 +1,4 @@ -.. _tutorial4: +.. _intro-tutorial4: ================= Finishing the job diff --git a/scrapy/trunk/docs/topics/item-pipeline.rst b/scrapy/trunk/docs/topics/item-pipeline.rst index 3b787ff43..408435b93 100644 --- a/scrapy/trunk/docs/topics/item-pipeline.rst +++ b/scrapy/trunk/docs/topics/item-pipeline.rst @@ -21,10 +21,13 @@ Writing your own item pipeline Writing your own item pipeline is easy. Each item pipeline component is a single Python class that must define the following method: -.. method:: process_item(request, spider) +.. method:: process_item(domain, response, item) -``request`` is a Request object -``spider`` is a BaseSpider object +``domain`` is a string with the domain of the spider which scraped the item + +``response`` is a :class:`scrapy.http.Response` with the response where the item was scraped + +``item`` is a :class:`scrapy.item.ScrapedItem` with the item scraped This method is called for every item pipeline component and must either return a ScrapedItem (or any descendant class) object or raise a :exception:`DropItem`