RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Updated scrapy overview 

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40549
This commit is contained in:
elpolilla 2008-12-26 12:21:53 +00:00
parent 4ee3626763
commit 867ee20c66
5 changed files with 108 additions and 93 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
sites/scrapy.org/docs/sources/intro/index.rst
View File

@ -4,5 +4,7 @@ Introduction
.. toctree::
:maxdepth: 1
overview
overview/overview
overview/spiders
overview/selectors
install

30
sites/scrapy.org/docs/sources/intro/overview/overview.rst Normal file
View File

@ -0,0 +1,30 @@
.. _overview:
Overview
========
Scrapy is a framework designed for retrieving information from websites.
The basic idea of scrapy is to be a robot that goes through websites, crawling pages, and extracting information from them.
The framework is formed by components that take care of different activities.
These components are basically:
* :ref:`spiders`
* :ref:`selectors`
* Items
* Adaptors
Features
--------
Scrapy includes many interesting features that make the scraping process much more easier and faster. These include:
* Asynchronous crawling/parsing on top of the Twisted framework.
* Easily configurable crawling through sets of rules.
* Ability for parsing HTML, XML, and CSV files.
* Media pipeline useful for scraping items with images or any other media files.
* *Very* extensible thanks to pipelines, middlewares, downloader-middlewares, and extensions.
* Automatic handling of compression, cache, cookies, authentication and more through already-included middlewares.
* Interactive scraping shell console, very useful for developing.

103
sites/scrapy.org/docs/sources/intro/overview.rst → sites/scrapy.org/docs/sources/intro/overview/selectors.rst
View File

@ -1,77 +1,4 @@
Overview
========
Scrapy is a framework designed for retrieving information from websites.
The basic idea of scrapy is to be a bot that goes through websites, getting pages, extracting links and items (which can be any kind of information).
The framework is formed by components that take care of different activities.
These components are *basically*:
* *Core*
* *Spiders*
* *Items*
* *Selectors*
* *Adaptors*
Core
----
Scrapy's core is the most important component (obviously) and the one in charge of getting/receiving information, and delivering it to your spiders (among many other things).
However, we won't talk about the core here, because it doesn't really concern anyone who wants to write spiders, so we'll move on.
Spiders
-------
We'll start off by the spiders because they're the ones that actually use the other components, and they are used themselves by scrapy's core, so they must be the first for you to know about.
Spiders are little programs, let's say, whose purpose is to scrape information from html pages or other data sources. Having said that, it's obvious that their process is something like:
1. Request for information and receive it.
2. Find what you were looking for in the response you got.
3. Adapt it (or not).
4. Store it.
Of course, you won't have to deal with sending requests and receiving responses yourself. Scrapy will do that job for you, so that you can concentrate in the one thing that probably brought you here: the information.
In order to tell Scrapy how do you want to handle your information, you must define a Spider. This is done by making a class that heredates from BaseSpider.
BaseSpider is a really basic class (as its name suggests), so it doesn't bring you much functionality for scraping HTML pages (since they require you to follow links, and differentiate many layouts), but it may be useful in case you're doing something totally different, or in case you want to write your own way of crawling.
In other words, BaseSpider is the most simple spider.
It has only these attributes:
* domain_name: which indicates the domain of the website you'd like to scrape in a string.
* start_urls: a list containing the urls you'd like to set as your enter point to the website.
* download_delay: an (optional) delay in seconds to wait before sending each request.
And these two methods:
* init_domain: it's called as soon as the spider is loaded, and it could be useful for initializing stuff.
* parse: this is the most important part of the spider, and it's where any response arrives.
So now you should figure out how spiders work, but just to make it sure, I'll show you an example::
from scrapy import log # This module is very useful for printing debug information
from scrapy.spider import BaseSpider
class MySpider(BaseSpider):
domain_name = 'http://www.mywebsite.com'
start_urls = [
'http://www.mywebsite.com/1.html',
'http://www.mywebsite.com/2.html',
'http://www.mywebsite.com/3.html',
]
def parse(self, response):
log.msg('Hey! A response from %s has just arrived!' % response.url)
return []
One **VERY** important thing that I didn't mention before, is the fact that the *parse* method must **always** return a list. Always!
This list may contain either Requests and/or Items (we'll talk about them right now, don't worry).
Now, anybody can fetch web pages -you'll think-, and it's true, you won't get too far by just dealing with that.
And that's when the selectors appear...
.. _selectors:
Selectors
---------
@ -79,17 +6,19 @@ Selectors are *the* way you have to extract information from documents. They ret
Currently there are two kinds of selectors, HtmlXPathSelectors, and XmlXPathSelectors. Both work in the same way; they are first instanciated with a response, for example::
hxs = HtmlXPathSelector(response)
hxs = HtmlXPathSelector(response) # an HTML selector
xxs = XmlXPathSelector(response) # an XML selector
Now, before going on with selectors, I must tell you about a pretty cool feature that Scrapy has, and which you'll surely find very useful whenever you're writing spiders.
This feature is the Scrapy shell, and you can use it by calling your project manager with the 'shell' argument; something like::
[user@host ~/myproject]$ ./scrapy-manager.py shell
[user@host ~/myproject]$ ./scrapy-manager.py shell <url>
Notice that you'll have to install IPython in order to use this feature, but believe me that it worths it; the shell is **very** useful.
With the shell you can simulate parsing a webpage, either by calling "scrapy-manager shell" with an url as an additional parameter, or by using the shell's 'get' command, which tries to retreive the given url, and fills in the 'response' variable with the result.
With the shell you can simulate parsing a webpage, either by calling "scrapy-manager shell" with an url as an additional parameter, or by using the shell's 'get' command, which tries
to retreive the given url, and fills in the 'response' variable with the result.
Ok, so now let's use the shell to show you a bit how do selectors work.
We'll use an example page located in Scrapy's site (http://docs.scrapy.org/examples/sample1.htm), whose markup is::
@ -126,7 +55,7 @@ Where 'r' is the object that scrapy already created for you containing the given
But anyway, we'll stick to the selectors scrapy already created for us, and more specifically, the HtmlXPathSelector (since we're working with an html document right now).
So let's try some expressions::
Let's try some expressions::
# The title
In [1]: hxs.x('//title/text()')
@ -142,7 +71,7 @@ So let's try some expressions::
# Image links
In [4]: hxs.x('//a[contains(@href, "image")]/@href').extract()
Out[4]:
Out[4]:
[u'image1.html',
u'image2.html',
u'image3.html',
@ -151,7 +80,7 @@ So let's try some expressions::
# Image thumbnails
In [5]: hxs.x('//a[contains(@href, "image")]/img/@src').extract()
Out[5]:
Out[5]:
[u'image1_thumb.jpg',
u'image2_thumb.jpg',
u'image3_thumb.jpg',
@ -160,7 +89,7 @@ So let's try some expressions::
# Image names
In [6]: hxs.x('//a[contains(@href, "image")]/text()').re(r'Name:\s*(.*)')
Out[6]:
Out[6]:
[u'My image 1',
u'My image 2',
u'My image 3',
@ -175,7 +104,7 @@ You can apply an x() call to any node you have, which means that you can join di
In [10]: links = hxs.x('//a[contains(@href, "image")]')
In [11]: links.extract()
Out[11]:
Out[11]:
[u'<a href="image1.html">Name: My image 1 <br><img src="image1_thumb.jpg"></a>',
u'<a href="image2.html">Name: My image 2 <br><img src="image2_thumb.jpg"></a>',
u'<a href="image3.html">Name: My image 3 <br><img src="image3_thumb.jpg"></a>',
@ -198,10 +127,8 @@ There are some things to keep in mind here:
2. x() calls are relative to the node your standing on, so selector.x('body/div[@id="mydiv"]') equals selector.x('body').x('div[@id="mydiv"]').
3. The extract() method *always* returns a list, even if it contains only one element. Don't forget that.
You may also have noticed that I've used another method up there; the re() method.
| You may also have noticed that I've used another method up there; the re() method.
| This one is very useful when the data extracted by XPath is not enough and you *have to* (remember to not abuse of regexp) make an extra parsing of the information you've got.
| In this cases, you just apply the re() method over any XPathSelector/XPathSelectorList you have with a compiled regexp pattern as the only argument, or a string with the pattern to be compiled.
| Remember that the re() method *always* returns an already extracted list, which means that you can't go back to a node from the result of a re() call (which is actually pretty obvious).
This one is very useful when the data extracted by XPath is not enough and you *have to* (remember to not abuse of regexp) make an extra parsing of the information you've got.
In this cases, you just apply the re() method over any XPathSelector/XPathSelectorList you have with a regexp compile pattern as the only argument, or a string with the pattern to be compiled.
Remember that the re() method *always* returns a list, which means that you can't go back to a node from the result of a re() call (which is actually pretty obvious).

56
sites/scrapy.org/docs/sources/intro/overview/spiders.rst Normal file
View File

@ -0,0 +1,56 @@
.. _spiders:
Spiders
-------
We'll start off by the spiders because they're the ones that actually use the other components, and they are used themselves by scrapy's core, so they must be the first for you to know about.
Spiders are little programs, let's say, whose purpose is to scrape information from html pages or other data sources. Having said that, it's obvious that their process is something like:
1. Request for information and receive it.
2. Find what you were looking for in the response you got.
3. Adapt it (or not).
4. Store it.
Of course, you won't have to deal with sending requests and receiving responses yourself. Scrapy will do that job for you, so that you can concentrate in the one thing that probably
brought you here: the information.
In order to tell Scrapy how do you want to handle your information, you must define a Spider. This is done by making a class that inherits from BaseSpider.
BaseSpider is a really basic class (as its name suggests), so it doesn't bring you much functionality for scraping HTML pages (since they require you to follow links,
and differentiate many layouts), but it may be useful in case you're doing something totally different, or in case you want to write your own way of crawling.
In other words, BaseSpider is the most simple available spider.
It has only these three attributes:
* domain_name: which indicates in a string the domain of the website you'd like to scrape.
* start_urls: a list containing the URLs you'd like to set as your entry point to the website.
* download_delay: an (optional) delay in seconds to wait before sending each request.
And these two methods:
* init_domain: it's called as soon as the spider is loaded, and it could be useful for initializing stuff.
* parse: this is the most important part of the spider, and it's where any response arrives.
So now you should figure out how spiders work, but just to make it sure, I'll show you an example::
from scrapy import log # This module is very useful for printing debug information
from scrapy.spider import BaseSpider
class MySpider(BaseSpider):
domain_name = 'http://www.mywebsite.com'
start_urls = [
'http://www.mywebsite.com/1.html',
'http://www.mywebsite.com/2.html',
'http://www.mywebsite.com/3.html',
]
def parse(self, response):
log.msg('Hey! A response from %s has just arrived!' % response.url)
return []
One **VERY** important thing that I didn't mention before, is the fact that the *parse* method must **always** return a list. Always!
This list may contain either Requests and/or Items (we'll talk about them later, don't worry).
Now, anybody can fetch web pages -you'll think-, and it's true, you won't get too far by just dealing with that.
And that's when the selectors make their appeareance...

8
sites/scrapy.org/docs/sources/tutorial/tutorial3.rst
View File

@ -6,7 +6,7 @@ Scraping our data
We will now browse a page containing links to websites stored in the directory (e.g. http://www.google.com/Top/Arts/Awards/) and see how can we extract
the information we need with XPath.
As I said before, you'll need FireBug for this.
As I said before, you'll need FireBug for this task.
|
|
@ -16,7 +16,7 @@ As I said before, you'll need FireBug for this.
| As you can see, this page's markup is not very descriptive (there are no id or name attributes, or anything that identifies the links uniquely),
so the ranking bars could be a nice reference at the moment of selecting the desired area with an XPath expression.
| After using FireBug, we can see that each link is inside a *td* tag, which is itself inside a *tr* tag that also contains the link's ranking bar.
| After using FireBug, we can see that each link is inside a *td* tag, which is itself inside a *tr* tag that also contains the link's ranking bar (in another *td*).
| So we could find the ranking bar; then from it, find its parent (the *tr*), and then finally, the link's *td* (which contains the data we want to scrape).
|
| We loaded the page in the Scrapy shell (very useful for doing this), and tried an XPath expression in order to find the links, which actually worked.
@ -24,7 +24,7 @@ As I said before, you'll need FireBug for this.
(the ranking bar's *td* tag), and then "return the *font* tag of each following *td* sibling that it has" (the link's *td* tag).
| Of course, this may not be the only way to get there (usually there are several expressions that get you to the same place), but it's quite good
for this case.
| Another approach could be to find any *font* tags that have that grey colour of the links, but I prefer to use the first one because it wouldn't be
| Another approach could be, for example, to find any *font* tags that have that grey colour of the links, but I prefer to use the first one because it wouldn't be
so strange if there were other tags with the same colour.
Anyway, having said that, a possible *parse_category* could be::
@ -68,5 +68,5 @@ are handled different than others (in fact, it *will* happen once you scrape mor
The rest of the code is quite self-explanatory. The *attribute* method sets the item's attributes, and the items themselves are put into a list that we'll return to Scrapy's engine.
One simple (although important) thing to remember here is that you must always return a list that contains either items, requests, or both, but always as a list.
One simple (although important) thing to remember here is that you must always return a list that contains either items, requests, or both, but always inside a list.