updated tutorial to reflect project's structure change

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40820
This commit is contained in:
Ismael Carnales 2009-02-05 12:14:49 +00:00
parent f7dac0e449
commit 751a844e36
4 changed files with 14 additions and 22 deletions

View File

@ -17,16 +17,11 @@ For starting a new project, enter the directory where you'd like your project to
As long as Scrapy is well installed and the path is set, this should create a directory called "google" containing the following files:
* *scrapy-ctl.py* - the project's control script. It's used for running the different tasks (like "genspider", "crawl" and "parse"). We'll talk more about this later.
* *scrapy_settings.py* - the project's settings file.
* *items.py* - were you define the different kinds of items you're going to scrape.
* *spiders* - directory where you'll later place your spiders.
* *templates* - directory containing some templates for newly created spiders, and where you can put your own.
| Ok, now that you have your project's structure defined, the last thing to do is to set your PYTHONPATH to your project's directory.
| You can do this by adding this to your .bashrc file:
::
$ export PYTHONPATH=/path/to/your/project
* *google/* - the projects' actual python module, you'll import your code from here.
* *google/items.py* - were you define the different kinds of items you're going to scrape.
* *google/pipelines.py* - were you define your item pipelines.
* *google/settings.py* - the project's settings file.
* *google/spiders/* - directory where you'll later place your spiders.
* *google/templates/* - directory containing some templates for newly created spiders, and where you can put your own.
Now you can continue with the next part of the tutorial: :ref:`intro-tutorial2`.

View File

@ -8,13 +8,13 @@ Ok, the time to write our first spider has come. Make sure that you're standing
./scrapy-ctl.py genspider google_directory google.com
This should create a file called google_directory.py under the *spiders* directory looking like this::
This should create a file called google_directory.py under the *google/spiders* directory looking like this::
# -*- coding: utf8 -*-
import re
from scrapy.xpath import HtmlXPathSelector
from scrapy.item import ScrapedItem
from google.items import GoogleScrapedItem
from scrapy.link.extractors import RegexLinkExtractor
from scrapy.contrib.spiders import CrawlSpider, Rule
@ -28,7 +28,7 @@ This should create a file called google_directory.py under the *spiders* directo
def parse_item(self, response):
xs = HtmlXPathSelector(response)
i = ScrapedItem()
i = GoogleScrapedItem()
#i.attribute('site_id', xs.x('//input[@id="sid"]/@value'))
#i.attribute('name', xs.x('//div[@id="name"]'))
#i.attribute('description', xs.x('//div[@id="description"]'))
@ -95,7 +95,7 @@ Until now, our spider would look something like::
# -*- coding: utf8 -*-
from scrapy.xpath import HtmlXPathSelector
from scrapy.item import ScrapedItem
from google.items import GoogleScrapedItem
from scrapy.link.extractors import RegexLinkExtractor
from scrapy.contrib.spiders import CrawlSpider, Rule

View File

@ -35,7 +35,7 @@ Anyway, having said that, a possible *parse_category* could be::
links = hxs.x('//td[descendant::a[contains(@href, "#pagerank")]]/following-sibling::td/font')
for link in links:
item = ScrapedItem()
item = GoogleScrapedItem()
adaptor_pipe = [adaptors.extract, adaptors.Delist(''), adaptors.strip]
item.set_adaptors({
'name': adaptor_pipe,

View File

@ -16,7 +16,7 @@ Let's see how would our spider end up looking like after applying this change::
# -*- coding: utf8 -*-
from scrapy.xpath import HtmlXPathSelector
from scrapy.item import ScrapedItem
from google.items import GoogleScrapedItem
from scrapy.contrib import adaptors
from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.link.extractors import RegexLinkExtractor
@ -40,7 +40,7 @@ Let's see how would our spider end up looking like after applying this change::
links = hxs.x('//td[descendant::a[contains(@href, "#pagerank")]]/following-sibling::td/font')
for link in links:
item = ScrapedItem()
item = GoogleScrapedItem()
adaptor_pipe = [adaptors.extract, adaptors.Delist(''), adaptors.strip]
item.set_adaptors({
'name': adaptor_pipe,
@ -60,11 +60,8 @@ Let's see how would our spider end up looking like after applying this change::
With this code, our spider will crawl over Google's directory, and save each
link's name, description, and url to a file called 'scraped_items.csv'.
link's name, description, and url to a file called 'scraped_items.csv'::
Cool, huh?
::
./scrapy-ctl.py crawl google.com
This is the end of the tutorial. If you'd like to know more about Scrapy and its use, please read the rest of the documentation.