mirror of https://github.com/scrapy/scrapy.git
Several documentation changes:
- merged (and updated) new tutorial from proposed doc - striped old tutorial and created new firebug topic - added topic about useful third firefox add-ons - rearranged main documentation index - several assorted documentation fixes --HG-- rename : scrapy/trunk/docs/proposed/tutorial.rst => scrapy/trunk/docs/intro/tutorial.rst rename : scrapy/trunk/docs/intro/tutorial/scrot1.png => scrapy/trunk/docs/topics/_images/firebug1.png rename : scrapy/trunk/docs/intro/tutorial/scrot2.png => scrapy/trunk/docs/topics/_images/firebug2.png rename : scrapy/trunk/docs/intro/tutorial/scrot3.png => scrapy/trunk/docs/topics/_images/firebug3.png extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401038
This commit is contained in:
Pablo Hoffman
parent
0fb3ee16d8
commit
7c66f1739d
|
|
@ -27,7 +27,7 @@ Does Scrapy work with Python 3.0?
|
|||
---------------------------------
|
||||
|
||||
No, and there are no plans to port Scrapy to Python 3.0 yet. At the moment
|
||||
Scrapy requires Python 2.5 or 2.6.
|
||||
Scrapy works with Python 2.5 or 2.6.
|
||||
|
||||
Does Scrapy "stole" X from Django?
|
||||
----------------------------------
|
||||
|
|
|
|||
|
|
@ -1,29 +1,75 @@
|
|||
.. _index:
|
||||
|
||||
==============================
|
||||
Scrapy |version| documentation
|
||||
================================
|
||||
==============================
|
||||
|
||||
Welcome! This is the documentation for Scrapy |version|, last updated on |today|.
|
||||
Welcome! This is the documentation for Scrapy, a screen scraping framework for
|
||||
Python.
|
||||
|
||||
For more information visit the `Scrapy homepage <http://scrapy.org>`_.
|
||||
For more information about the project visit the `Scrapy homepage
|
||||
<http://scrapy.org>`_.
|
||||
|
||||
Documentation Contents:
|
||||
Getting started
|
||||
===============
|
||||
|
||||
This section helps you get familiarized with Scrapy and its basic concepts.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
intro/index
|
||||
topics/index
|
||||
ref/index
|
||||
|
||||
misc/index
|
||||
Using Scrapy
|
||||
============
|
||||
|
||||
This section explains all key concepts of Scrapy.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
:maxdepth: 2
|
||||
|
||||
topics/index
|
||||
|
||||
API reference
|
||||
=============
|
||||
|
||||
This section documents the :ref:`Scrapy API <ref-index>`. See also
|
||||
:ref:`misc-api-stability`.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
ref/index
|
||||
|
||||
Frequently asked questions
|
||||
==========================
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
faq
|
||||
|
||||
Experimental features
|
||||
=====================
|
||||
|
||||
This section documents experimental features that may become stable in future
|
||||
Scrapy releases, but whose API is not yet stable. Use them with caution, and
|
||||
subscribe to the `mailing lists <http://scrapy.org/community/>`_ to be notified
|
||||
of any changes.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
topics/adaptors
|
||||
|
||||
Proposed documentation
|
||||
======================
|
||||
|
||||
This section contains documentation which is not yet completed, or which
|
||||
overlaps with existing documentation and needs to be manually merged. It may
|
||||
also contain outdated information, as it's not revised so frequently.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
proposed/index
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@ might help you.
|
|||
|
||||
overview
|
||||
install
|
||||
tutorial/index
|
||||
tutorial
|
||||
|
||||
.. seealso::
|
||||
|
||||
|
|
|
|||
|
|
@ -1,8 +1,11 @@
|
|||
.. _intro-install:
|
||||
|
||||
============
|
||||
Installation
|
||||
============
|
||||
==================
|
||||
Installation guide
|
||||
==================
|
||||
|
||||
This document describes how to install Scrapy in Linux, Windows and Mac OS X
|
||||
systems.
|
||||
|
||||
.. highlight:: sh
|
||||
|
||||
|
|
|
|||
|
|
@ -1,21 +1,23 @@
|
|||
.. _tutorial:
|
||||
.. _intro-tutorial:
|
||||
|
||||
===============
|
||||
Scrapy Tutorial
|
||||
===============
|
||||
|
||||
In this tutorial, we'll assume that Scrapy is already installed in your system,
|
||||
if not see :ref:`intro-install`.
|
||||
In this tutorial, we'll assume that Scrapy is already installed in your system.
|
||||
If that's not the case see :ref:`intro-install`.
|
||||
|
||||
We are going to use `Open directory project (dmoz) <http://www.dmoz.org/>`_ as
|
||||
our example domain to scrape.
|
||||
|
||||
This tutorial will introduce you to this tasks:
|
||||
This tutorial will walk you through through these tasks:
|
||||
|
||||
* Creating a project
|
||||
* Defining the Items you will extract
|
||||
* Writing a spider to crawl a site and extract Items
|
||||
* Write an Item Pipeline to store the extracted Items
|
||||
1. Creating a new Scrapy project
|
||||
2. Defining the Items you will extract
|
||||
3. Writing a :ref:`spider <topics-spiders>` to crawl a site and extract
|
||||
:ref:`Items <topics-items>`
|
||||
4. Writing an :ref:`Item Pipeline <topics-item-pipeline>` to store the
|
||||
extracted Items
|
||||
|
||||
Creating a project
|
||||
==================
|
||||
|
|
@ -73,10 +75,11 @@ Our first Spider
|
|||
Spiders are user written classes to scrape information from a domain (or group
|
||||
of domains).
|
||||
|
||||
They define an initial set of URLs to download, and how to parse the downloaded contents in the search for data (Items) or more URLs to follow.
|
||||
They define an initial list of URLs to download, how to follow links, and how
|
||||
to parse the contents of those pages to extract :ref:`items <topics-items>`.
|
||||
|
||||
To create a Spider, you must subclass :class:`scrapy.spider.BaseSpider`, and
|
||||
then define the three main, mandatory, attributes:
|
||||
define the three main, mandatory, attributes:
|
||||
|
||||
* :attr:`~scrapy.spider.BaseSpider.domain_name`: identifies the Spider. It must
|
||||
be unique, that is, you can't set the same domain name for different Spiders.
|
||||
|
|
@ -86,16 +89,20 @@ then define the three main, mandatory, attributes:
|
|||
listed here. The subsequent URLs will be generated successively from data
|
||||
contained in the start URLs.
|
||||
|
||||
* :meth:`~scrapy.spider.BaseSpider.parse` is the callback method of the spider.
|
||||
This means that each time a URL is retrieved, the downloaded data (Response)
|
||||
will be passed to this method.
|
||||
* :meth:`~scrapy.spider.BaseSpider.parse` is a method of the spider, which will
|
||||
be called with the downloaded :class:`~scrapy.http.Response` object of each
|
||||
start URL. The response is passed to the method as the first and only
|
||||
argument.
|
||||
|
||||
This method is responsible for parsing the response data and extracting
|
||||
scraped data (as scraped items) and more URLs to follow.
|
||||
|
||||
The :meth:`~scrapy.spider.BaseSpider.parse` method is in charge of processing
|
||||
the response and returning scraped data and or more URLs to follow, because of
|
||||
this, the method must always return a list or at least an empty one.
|
||||
the response and returning scraped data (as :class:`~scrapy.item.ScrapedItem`
|
||||
objects) and more URLs to follow (as :class:`~scrapy.http.Request` objects).
|
||||
|
||||
This is the code for our first Spider, save it in a file named
|
||||
``dmoz_spider.py`` inside ``dmoz/spiders`` directory::
|
||||
``dmoz_spider.py`` under the ``dmoz/spiders`` directory::
|
||||
|
||||
from scrapy.spider import BaseSpider
|
||||
|
||||
|
|
@ -113,19 +120,15 @@ This is the code for our first Spider, save it in a file named
|
|||
|
||||
SPIDER = DmozSpider()
|
||||
|
||||
.. warning::
|
||||
|
||||
When creating spiders, be sure not to name them equal to the project's name
|
||||
or you won't be able to import modules from your project in your spider!
|
||||
|
||||
Crawling
|
||||
--------
|
||||
|
||||
To put our spider to work, go to the project's top level directory and run::
|
||||
|
||||
./scrapy-ctl.py crawl dmoz.org
|
||||
python scrapy-ctl.py crawl dmoz.org
|
||||
|
||||
The ``crawl dmoz.org`` subcommand runs the spider for the ``dmoz.org`` domain, you'll get an output like this::
|
||||
The ``crawl dmoz.org`` subcommand runs the spider for the ``dmoz.org`` domain,
|
||||
you'll get an output like this::
|
||||
|
||||
[-] Log opened.
|
||||
[dmoz] INFO: Enabled extensions: TelnetConsole, WebConsole
|
||||
|
|
@ -145,8 +148,8 @@ The ``crawl dmoz.org`` subcommand runs the spider for the ``dmoz.org`` domain, y
|
|||
Pay attention to the lines labeled ``[dmoz/dmoz.org]``, which corresponds to
|
||||
our spider identified by the domain "dmoz.org". You can see a log line for each
|
||||
URL defined in ``start_urls``. Because these URLs are the starting ones, they
|
||||
have no referrers, and this condition is indicated at the end of the log line,
|
||||
where it says ``from <None>``.
|
||||
have no referrers, which is shown at the end of the log line, where it says
|
||||
``from <None>``.
|
||||
|
||||
But more interesting, as our ``parse`` method instructs, two files have been
|
||||
created: *Books* and *Resources*, with the content of both URLs.
|
||||
|
|
@ -158,8 +161,9 @@ Scrapy creates :class:`scrapy.http.Request` objects for each URL in the
|
|||
``start_urls`` attribute of the Spider, and assigns them the ``parse`` method of
|
||||
the spider as their callback function.
|
||||
|
||||
These Requests are scheduled, then executed, and :class:`scrapy.http.Response`
|
||||
objects are returned to the generator of the Requests.
|
||||
These Requests are scheduled, then executed, and a :class:`scrapy.http.Response`
|
||||
objects are returned and then fed to the spider, through the
|
||||
:meth:`~scrapy.spider.BaseSpider.parse` method.
|
||||
|
||||
Extracting Items
|
||||
----------------
|
||||
|
|
@ -167,61 +171,73 @@ Extracting Items
|
|||
Introduction to Selectors
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
In order to extract information from web pages Scrapy adopted `XPath
|
||||
<http://www.w3.org/TR/xpath>`_, a language for finding information in a XML
|
||||
document navigating trough its elements and attributes.
|
||||
There are several ways to extract data from web pages, Scrapy uses a mechanism
|
||||
based on `XPath`_ expressions called :ref:`XPath selectors <topics-selectors>`.
|
||||
For more information about selectors and other extraction mechanisms see the
|
||||
:ref:`XPath selectors documentation <topics-selectors>`.
|
||||
|
||||
Here are some examples of XPath queries and their corresponding results:
|
||||
.. _XPath: http://www.w3.org/TR/xpath
|
||||
|
||||
* ``/html/head/title``: Will give you the ``title`` node of the document.
|
||||
* ``/html/head/title/text()``: Will give you the text inside the ``title`` node of the document.
|
||||
* ``//td``: Will select all the ``td`` elements.
|
||||
* ``//div[@class="queryMe"]``: Will select all the ``div`` elements with ``class
|
||||
= queryMe``.
|
||||
Here are some examples of XPath expressions and their meanings:
|
||||
|
||||
This are really simple examples of what you can do with XPath, we strongly
|
||||
suggest you to follow this `XPath tutorial
|
||||
<http://www.w3schools.com/XPath/default.asp>`_ before continuing.
|
||||
* ``/html/head/title``: selects the ``<title>`` element, inside the ``<head>``
|
||||
element of a HTML document
|
||||
|
||||
Scrapy defines a class :class:`~scrapy.xpath.XPathSelector`, that comes in two
|
||||
flavours, :class:`~scrapy.xpath.HtmlXPatSelector` (for HTML) and
|
||||
:class:`~scrapy.xpath.XmlXPathSelector` (for XML). In order to use them you
|
||||
must instantiate the desired class with a :ref:`Response <request-response>`
|
||||
object.
|
||||
* ``/html/head/title/text()``: selects the text inside the aforementioned
|
||||
``<title>`` element.
|
||||
|
||||
* ``//td``: selects all the ``<td>`` elements
|
||||
|
||||
* ``//div[@class="mine"]``: selects all ``div`` elements which contain an
|
||||
attribute ``class="mine"``
|
||||
|
||||
These are just a couple of simple examples of what you can do with XPath, but
|
||||
XPath expression are indeed much more powerful. To learn more about XPath we
|
||||
recommend `this XPath tutorial <http://www.w3schools.com/XPath/default.asp>`_.
|
||||
|
||||
For working with XPaths, Scrapy provides a :class:`~scrapy.xpath.XPathSelector`
|
||||
class, which comes in two flavours, :class:`~scrapy.xpath.HtmlXPatSelector`
|
||||
(for HTML data) and :class:`~scrapy.xpath.XmlXPathSelector` (for XML data). In
|
||||
order to use them you must instantiate the desired class with a
|
||||
:class:`~scrapy.http.Response` object.
|
||||
|
||||
You can see selectors as objects that represents nodes in the document
|
||||
structure. So, the first instantiated selectors are associated to the root
|
||||
node, or the entire document.
|
||||
|
||||
Selectors have three methods: ``x``, ``extract`` and ``re``.
|
||||
Selectors have three methods (click on the method to see the complete API
|
||||
documentation).
|
||||
|
||||
* :meth:`~scrapy.xpath.XPathSelector.x`: returns a list of selectors, each of
|
||||
them representing the nodes selected by the xpath expression given as
|
||||
argument.
|
||||
|
||||
* :meth:`~scrapy.xpath.XPathSelector.extract`: returns a unicode string with
|
||||
the data selected by the XPath selector.
|
||||
|
||||
* :meth:`~scrapy.xpath.XPathSelector.re`: returns a list unicode strings
|
||||
extracted by applying the regular expression given as argument.
|
||||
|
||||
* ``x``: returns a list of selectors, each of them representing the nodes
|
||||
gotten in the xpath expression given as parameter.
|
||||
* ``extract``: actually extracts the data contained in the node. Does not
|
||||
receive parameters.
|
||||
* ``re``: returns a list of results of a regular expression given as parameter.
|
||||
|
||||
Trying Selectors in the Shell
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
To illustrate the use of Selectors we're going to use the built-in shell of
|
||||
Scrapy, notice that in order to use this feature you must have IPython (an
|
||||
extended Python console) installed on your system.
|
||||
To illustrate the use of Selectors we're going to use the built-in :ref:`Scrapy
|
||||
shell <topics-shell>`, which also requires IPython (an extended Python console)
|
||||
installed on your system.
|
||||
|
||||
To start a shell you must go to the project's top level directory and run::
|
||||
|
||||
./scrapy-ctl.py shell http://www.dmoz.org/Computers/Programming/Languages/Python/Books/
|
||||
python scrapy-ctl.py shell http://www.dmoz.org/Computers/Programming/Languages/Python/Books/
|
||||
|
||||
This is what the shell looks like::
|
||||
|
||||
[-] Log opened.
|
||||
Scrapy 0.7.0 - Interactive scraping console
|
||||
Welcome to Scrapy shell!
|
||||
Fetching <http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>...
|
||||
|
||||
[-] scrapy.management.web.WebConsole starting on 33227
|
||||
[-] scrapy.management.telnet.TelnetConsole starting on 42311
|
||||
Downloading URL... Done.
|
||||
------------------------------------------------------------------------------
|
||||
Available local variables:
|
||||
Available Scrapy variables:
|
||||
xxs: <class 'scrapy.xpath.selector.XmlXPathSelector'>
|
||||
url: http://www.dmoz.org/Computers/Programming/Languages/Python/Books/
|
||||
spider: <class 'dmoz.spiders.dmoz.OpenDirectorySpider'>
|
||||
|
|
@ -229,8 +245,8 @@ This is what the shell looks like::
|
|||
item: <class 'scrapy.item.models.ScrapedItem'>
|
||||
response: <class 'scrapy.http.response.html.HtmlResponse'>
|
||||
Available commands:
|
||||
get <url>: Fetches an url and updates all variables.
|
||||
scrapehelp: Prints this help.
|
||||
get [url]: Fetch a new URL or re-fetch current Request
|
||||
shelp: Prints this help.
|
||||
------------------------------------------------------------------------------
|
||||
Python 2.6.1 (r261:67515, Dec 7 2008, 08:27:41)
|
||||
Type "copyright", "credits" or "license" for more information.
|
||||
|
|
@ -243,12 +259,13 @@ This is what the shell looks like::
|
|||
|
||||
In [1]:
|
||||
|
||||
After the shell loads, it will put the result of the request action for the
|
||||
given URL in a ``response`` variable, so if you enter ``response.body`` the
|
||||
downloaded data will be printed on the screen.
|
||||
After the shell loads, you will have the response fetched in a local
|
||||
``response`` variable, so if you type ``response.body`` you will see the body
|
||||
of the response, or you can ``response.headers`` to see its headers.
|
||||
|
||||
The shell has also instantiated for two selectors with this respose as an
|
||||
initialization parameter, so let's try them::
|
||||
The shell also instantiates two selectors, one for HTML (in the ``hxs``
|
||||
variable) and one for XML (in the ``xxs`` variable)with this response. So let's
|
||||
try them::
|
||||
|
||||
In [1]: hxs.x('/html/head/title')
|
||||
Out[1]: [<HtmlXPathSelector (title) xpath=/html/head/title>]
|
||||
|
|
@ -265,16 +282,22 @@ initialization parameter, so let's try them::
|
|||
In [5]: hxs.x('/html/head/title/text()').re('(\w+):')
|
||||
Out[5]: [u'Computers', u'Programming', u'Languages', u'Python']
|
||||
|
||||
Actually extracting Items
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
Extracting the data
|
||||
^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Now, let's try to extract the sites information from the directory page.
|
||||
Now, let's try to extract some real information from those pages.
|
||||
|
||||
If you do a ``response.body`` in the console, look at the source code of the
|
||||
page or better yet use Firebug to inspect the page, you'll find that the sites
|
||||
part of the code is an ``ul`` tag, in fact the *second* ``ul`` tag.
|
||||
You could type ``response.body`` in the console, and inspect the source code to
|
||||
figure out the XPaths you need to use. However, inspecting the raw HTML code
|
||||
there could become a very tedious task. To make this an easier task, you can
|
||||
use some Firefox extensions like Firebug. For more information see
|
||||
:ref:`topics-firebug` and :ref:`topics-firefox`.
|
||||
|
||||
So we can select each ``li`` item belonging to the sites list with this code::
|
||||
After inspecting the page source you'll find that the web sites information
|
||||
is inside a ``<ul>`` element, in fact the *second* ``<ul>`` element.
|
||||
|
||||
So we can select each ``<li>`` element belonging to the sites list with this
|
||||
code::
|
||||
|
||||
hxs.x('//ul[2]/li')
|
||||
|
||||
|
|
@ -306,7 +329,6 @@ Let's add this code to our spider::
|
|||
from scrapy.spider import BaseSpider
|
||||
from scrapy.xpath.selector import HtmlXPathSelector
|
||||
|
||||
|
||||
class DmozSpider(BaseSpider):
|
||||
domain_name = "dmoz.org"
|
||||
start_urls = [
|
||||
|
|
@ -329,7 +351,7 @@ Let's add this code to our spider::
|
|||
Now try crawling the dmoz.org domain again and you'll see sites being printed
|
||||
in your output, run::
|
||||
|
||||
./scrapy-ctl.py crawl dmoz.org
|
||||
python scrapy-ctl.py crawl dmoz.org
|
||||
|
||||
Spiders are supposed to return their scraped data in the form of ScrapedItems,
|
||||
so to actually return the data we've scraped so far, the code for our Spider
|
||||
|
|
@ -340,7 +362,6 @@ should be like this::
|
|||
|
||||
from dmoz.items import DmozItem
|
||||
|
||||
|
||||
class DmozSpider(BaseSpider):
|
||||
domain_name = "dmoz.org"
|
||||
start_urls = [
|
||||
|
|
@ -362,29 +383,31 @@ should be like this::
|
|||
|
||||
SPIDER = DmozSpider()
|
||||
|
||||
Now doing a crawl on the dmoz.org domain yields DmozItems::
|
||||
Now doing a crawl on the dmoz.org domain yields ``DmozItem``'s::
|
||||
|
||||
[dmoz/dmoz.org] DEBUG: Scraped DmozItem({'title': [u'Text Processing in Python'], 'link': [u'http://gnosis.cx/TPiP/'], 'desc': [u' - By David Mertz; Addison Wesley. Book in progress, full text, ASCII format. Asks for feedback. [author website, Gnosis Software, Inc.]\n']}) in <http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>
|
||||
[dmoz/dmoz.org] DEBUG: Scraped DmozItem({'title': [u'XML Processing with Python'], 'link': [u'http://www.informit.com/store/product.aspx?isbn=0130211192'], 'desc': [u' - By Sean McGrath; Prentice Hall PTR, 2000, ISBN 0130211192, has CD-ROM. Methods to build XML applications fast, Python tutorial, DOM and SAX, new Pyxie open source XML processing library. [Prentice Hall PTR]\n']}) in <http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>
|
||||
|
||||
|
||||
Item Pipeline
|
||||
=============
|
||||
Storing the data (using an Item Pipeline)
|
||||
=========================================
|
||||
|
||||
After an item has been scraped by a Spider, it is sent to the Item Pipeline.
|
||||
After an item has been scraped by a Spider, it is sent to the :ref:`Item
|
||||
Pipeline <topics-item-pipeline>`.
|
||||
|
||||
The Item Pipeline is a set of user written Python classes that implement a
|
||||
simple method. They receive the Item, do an action upon it (like validating,
|
||||
checking for duplicates, store the item), and then decide if the Item continues
|
||||
trough the Pipeline or it's dropped.
|
||||
The Item Pipeline is a group of user written Python classes that implement a
|
||||
simple method. They receive an Item and perform an action over it (for example:
|
||||
validation, checking for duplicates, or storing it in a database), and then
|
||||
decide if the Item continues through the Pipeline or it's dropped and no longer
|
||||
processed.
|
||||
|
||||
In small projects like this we will use only one Item Pipeline that stores our
|
||||
Items.
|
||||
In small projects (like the one on this tutorial) we will use only one Item
|
||||
Pipeline that just stores our Items.
|
||||
|
||||
Like with the Item, a Pipeline placeholder has been set up for you in the
|
||||
project creation step, it's in ``dmoz/pipelines.py`` and looks like this::
|
||||
As with Items, a Pipeline placeholder has been set up for you in the project
|
||||
creation step, it's in ``dmoz/pipelines.py`` and looks like this::
|
||||
|
||||
# Define yours item pipelines here
|
||||
# Define your item pipelines here
|
||||
|
||||
class DmozPipeline(object):
|
||||
def process_item(self, domain, item):
|
||||
|
|
@ -404,5 +427,6 @@ example in a csv file::
|
|||
Finale
|
||||
======
|
||||
|
||||
This covers the basics of Scrapy, but they're a lot of features that haven't
|
||||
been mentioned. They'll be in further tutorials.
|
||||
This tutorial covers only the basics of Scrapy, but there's a lot of other
|
||||
features not mentioned here. We recommend you continue reading the section
|
||||
:ref:`topics-index`.
|
||||
|
|
@ -1,10 +0,0 @@
|
|||
========
|
||||
Tutorial
|
||||
========
|
||||
|
||||
.. toctree::
|
||||
|
||||
tutorial1
|
||||
tutorial2
|
||||
tutorial3
|
||||
tutorial4
|
||||
|
|
@ -1,55 +0,0 @@
|
|||
.. _intro-tutorial1:
|
||||
|
||||
======================
|
||||
Creating a new project
|
||||
======================
|
||||
|
||||
.. highlight:: sh
|
||||
|
||||
In this tutorial, we'll teach you how to scrape http://www.google.com/dirhp
|
||||
Google's web directory.
|
||||
|
||||
We'll assume that Scrapy is already installed in your system, if not see
|
||||
:ref:`intro-install`.
|
||||
|
||||
For starting a new project, enter the directory where you'd like your project
|
||||
to be located, and run::
|
||||
|
||||
$ scrapy-admin.py startproject googledir
|
||||
|
||||
As long as Scrapy is well installed and the path is set, this will create a
|
||||
``googledir`` directory with the following contents::
|
||||
|
||||
googledir/
|
||||
scrapy-ctl.py
|
||||
googledir/
|
||||
__init__.py
|
||||
items.py
|
||||
pipelines.py
|
||||
settings.py
|
||||
spiders/
|
||||
__init__.py
|
||||
templates/
|
||||
...
|
||||
|
||||
These are basically:
|
||||
|
||||
* ``scrapy-ctl.py``: the project's control script. It's used for running the
|
||||
different tasks (like "genspider", "crawl" and "parse"). We'll talk more
|
||||
about this later.
|
||||
|
||||
* ``googledir/``: the project's actual python module, you'll import your code from here.
|
||||
|
||||
* ``googledir/items.py``: were you define the different kinds of items you're going to scrape.
|
||||
|
||||
* ``googledir/pipelines.py``: were you define your item pipelines.
|
||||
|
||||
* ``googledir/settings.py``: the project's settings file.
|
||||
|
||||
* ``googledir/spiders/``: directory where you'll later place your spiders.
|
||||
|
||||
* ``googledir/templates/``: directory containing some templates for newly created
|
||||
spiders, and where you can put your own.
|
||||
|
||||
|
||||
Now you can continue with the next part of the tutorial: :ref:`intro-tutorial2`.
|
||||
|
|
@ -1,158 +0,0 @@
|
|||
.. _intro-tutorial2:
|
||||
|
||||
================
|
||||
Our first spider
|
||||
================
|
||||
|
||||
Ok, the time to write our first spider has come. Make sure that you're standing
|
||||
on your project's directory and run::
|
||||
|
||||
./scrapy-ctl.py genspider google_directory google.com
|
||||
|
||||
This should create a file called google_directory.py under ``googledir/spiders``
|
||||
directory looking like this::
|
||||
|
||||
# -*- coding: utf8 -*-
|
||||
import re
|
||||
|
||||
from scrapy.xpath import HtmlXPathSelector
|
||||
from scrapy.link.extractors import RegexLinkExtractor
|
||||
from scrapy.contrib.spiders import CrawlSpider, Rule
|
||||
from googledir.items import GoogledirItem
|
||||
|
||||
class GoogleDirectorySpider(CrawlSpider):
|
||||
domain_name = 'google.com'
|
||||
start_urls = ['http://www.google.com/']
|
||||
|
||||
rules = (
|
||||
Rule(RegexLinkExtractor(allow=(r'Items/', )), 'parse_item', follow=True),
|
||||
)
|
||||
|
||||
def parse_item(self, response):
|
||||
i = GoogledirItem()
|
||||
#xs = HtmlXPathSelector(response)
|
||||
#i.attribute('site_id', xs.x('//input[@id="sid"]/@value'))
|
||||
#i.attribute('name', xs.x('//div[@id="name"]'))
|
||||
#i.attribute('description', xs.x('//div[@id="description"]'))
|
||||
return [i]
|
||||
|
||||
SPIDER = GoogleDirectorySpider()
|
||||
|
||||
Now, let's explain a bit what this is all about.
|
||||
|
||||
As you may have noticed, the class that represents the spider is
|
||||
GoogleDirectorySpider, and it inherits from CrawlSpider.
|
||||
|
||||
This means that this spider will crawl over a website given some crawling
|
||||
rules, and parse the response you need according to your patterns, which are
|
||||
defined through the "rules" class attribute.
|
||||
|
||||
This attribute is nothing else but a tuple containing Rule objects. Each Rule
|
||||
defines a specific behaviour the spider will have while crawling the site.
|
||||
|
||||
Rule objects accept the following parameters (the ones between [ ] are optional):
|
||||
|
||||
* ``link_extractor`` - A LinkExtractor instance, which defines the crawling
|
||||
patterns for this Rule.
|
||||
|
||||
* ``callback`` - A callback to be called for each link extracted matching the
|
||||
previous link extractor.
|
||||
|
||||
* ``cb_kwargs`` - A dictionary of keyword arguments to be passed to the
|
||||
provided callback.
|
||||
|
||||
* ``follow`` - A boolean that determines if links are going to be extracted
|
||||
from responses matching this Rule or not.
|
||||
|
||||
* ``process_links`` - An optional callback for parsing the extracted links.
|
||||
|
||||
In this case, the Rule would mean something like "search for any url containing
|
||||
the string 'Items/', parse it with the 'parse_item' method, and try to extract
|
||||
more links from it".
|
||||
|
||||
Now, that's an example, so we must make our own Rule for our own spider.
|
||||
|
||||
But before that, we must set our start_urls to our real entry point (which is
|
||||
not actually Google's homepage).
|
||||
|
||||
So we replace that line with::
|
||||
|
||||
start_urls = ['http://www.google.com/dirhp']
|
||||
|
||||
Now it's the moment to surf that page, and see how can we do to extract data
|
||||
from it.
|
||||
|
||||
For this task is almost mandatory that you have Firefox FireBug extension,
|
||||
which allows you to browse through HTML markup in an easy and comfortable way.
|
||||
Otherwise you'd have to search for tags manually through the body, which can be
|
||||
*very* tedious.
|
||||
|
||||
.. image:: scrot1.png
|
||||
|
||||
What we see at first sight, is that the directory is divided in categories,
|
||||
which are also divided in subcategories.
|
||||
|
||||
However, it seems as if there are more subcategories than the ones being shown
|
||||
in this page, so we'll keep looking...
|
||||
|
||||
.. image:: scrot2.png
|
||||
|
||||
Right, this looks much more interesting. Not only subcategories themselves have
|
||||
more subcategories, but they have links to websites (which is in fact the
|
||||
purpose of the directory).
|
||||
|
||||
Now, there's basically one thing to take into account about the previous, and
|
||||
it's the fact that apparently, categories urls are always of the kind
|
||||
http://www.google.com/Category/Subcategory/Another_Subcategory (which is not
|
||||
very distinctive actually, but possible to use).
|
||||
|
||||
So, having said that, a possible rule set for the categories could be::
|
||||
|
||||
rules = (
|
||||
Rule(RegexLinkExtractor(allow=('google.com/[A-Z][a-zA-Z_/]+$', ), ),
|
||||
'parse_category',
|
||||
follow=True,
|
||||
),
|
||||
)
|
||||
|
||||
Basically, we told our Rule object to extract links that contain the string
|
||||
'google.com/' plus any capital letter, plus any letter, the '_' character or
|
||||
the '/'.
|
||||
|
||||
Also, we set our callback 'parse_category' for each of those crawled links, and
|
||||
decided to extract more links from them with follow=True.
|
||||
|
||||
Until now, our spider would look something like::
|
||||
|
||||
# -*- coding: utf8 -*-
|
||||
import re
|
||||
|
||||
from scrapy.xpath import HtmlXPathSelector
|
||||
from scrapy.link.extractors import RegexLinkExtractor
|
||||
from scrapy.contrib.spiders import CrawlSpider, Rule
|
||||
from googledir.items import GoogledirItem
|
||||
|
||||
class GoogleDirectorySpider(CrawlSpider):
|
||||
domain_name = 'google.com'
|
||||
start_urls = ['http://www.google.com/dirhp']
|
||||
|
||||
rules = (
|
||||
Rule(RegexLinkExtractor(allow=('google.com/[A-Z][a-zA-Z_/]+$',),),
|
||||
'parse_category',
|
||||
follow=True,
|
||||
),
|
||||
)
|
||||
|
||||
def parse_category(self, response):
|
||||
pass
|
||||
|
||||
SPIDER = GoogleDirectorySpider()
|
||||
|
||||
You can try crawling with this little code, by running::
|
||||
|
||||
./scrapy-ctl.py crawl google.com
|
||||
|
||||
and it will actually work, altough it won't do any parsing, since
|
||||
parse_category is not defined, and that's exactly what we're going to do in the
|
||||
next part of the tutorial: :ref:`intro-tutorial3`.
|
||||
|
||||
|
|
@ -1,150 +0,0 @@
|
|||
.. _intro-tutorial3:
|
||||
|
||||
=================
|
||||
Scraping our data
|
||||
=================
|
||||
|
||||
Before going to extraction we need to make a change to our container of scraped
|
||||
data called ``GoogledirItem`` and defined in ``googledir/items.py`` module::
|
||||
|
||||
from scrapy.contrib.item import RobustScrapedItem
|
||||
|
||||
class GoogledirItem(RobustScrapedItem):
|
||||
"""Directory website link"""
|
||||
|
||||
ATTRIBUTES = {
|
||||
'guid': basestring,
|
||||
'name': basestring,
|
||||
'url': basestring,
|
||||
'description': basestring,
|
||||
}
|
||||
|
||||
Be sure to inherit from RobustScrapedItem, not ScrapedItem.
|
||||
|
||||
We will now browse a page containing links to websites stored in the directory
|
||||
(e.g. http://www.google.com/Top/Arts/Awards/) and see how can we extract the
|
||||
information we need with XPath.
|
||||
|
||||
As I said before, you'll need FireBug for this task.
|
||||
|
||||
.. image:: scrot3.png
|
||||
|
||||
As you can see, this page's markup is not very descriptive (there are no id or
|
||||
name attributes, or anything that identifies the links uniquely), so the
|
||||
ranking bars could be a nice reference at the moment of selecting the desired
|
||||
area with an XPath expression.
|
||||
|
||||
After using FireBug, we can see that each link is inside a ``td`` tag, which is
|
||||
itself inside a ``tr`` tag that also contains the link's ranking bar (in another
|
||||
``td``).
|
||||
|
||||
So we could find the ranking bar; then from it, find its parent (the ``tr``),
|
||||
and then finally, the link's ``td`` (which contains the data we want to
|
||||
scrape).
|
||||
|
||||
We loaded the page in the Scrapy shell (very useful for doing this), and tried
|
||||
an XPath expression in order to find the links, which actually worked.
|
||||
|
||||
Basically, that expression would looks for the ranking bar's ``td`` tag:
|
||||
"find any ``td`` tag who has a descendant tag ``a`` whose ``href``
|
||||
attribute contains the string ``#pagerank``"
|
||||
|
||||
and then, the link's ``td`` tag:
|
||||
"return the ``font`` tag of each following ``td`` sibling that it has"
|
||||
|
||||
Of course, this may not be the only way to get there (usually there are several
|
||||
expressions that get you to the same place), but it's quite good for this case.
|
||||
|
||||
Another approach could be, for example, to find any ``font`` tags that have that
|
||||
grey colour of the links, but I prefer to use the first one because it wouldn't
|
||||
be so strange if there were other tags with the same colour.
|
||||
|
||||
Anyway, having said that, let's make a possible ``parse_category``:
|
||||
|
||||
First, modify the imports section in the spider code to look like this::
|
||||
|
||||
import re
|
||||
|
||||
from scrapy.xpath import HtmlXPathSelector
|
||||
from scrapy.link.extractors import RegexLinkExtractor
|
||||
from scrapy.contrib.spiders import CrawlSpider, Rule
|
||||
from scrapy.contrib_exp import adaptors
|
||||
from googledir.items import GoogledirItem
|
||||
|
||||
Then, put this code in the ``parse_category`` method::
|
||||
|
||||
from scrapy.contrib_exp import adaptors
|
||||
|
||||
def parse_category(self, response):
|
||||
# The selector we're going to use in order to extract data from the page
|
||||
hxs = HtmlXPathSelector(response)
|
||||
|
||||
# The path to website links in directory page
|
||||
links = hxs.x('//td[descendant::a[contains(@href, "#pagerank")]]/following-sibling::td/font')
|
||||
|
||||
# The list of functions to apply to an attribute before assigning its value
|
||||
adaptor_pipe = [adaptors.extract, adaptors.delist(''), adaptors.strip]
|
||||
adaptor_map = {
|
||||
'name': adaptor_pipe,
|
||||
'url': adaptor_pipe,
|
||||
'description': adaptor_pipe,
|
||||
}
|
||||
|
||||
for link in links:
|
||||
item = GoogledirItem()
|
||||
item.set_adaptors(adaptor_map)
|
||||
|
||||
item.attribute('name', link.x('a/text()'))
|
||||
item.attribute('url', link.x('a/@href'))
|
||||
item.attribute('description', link.x('font[2]/text()'))
|
||||
yield item
|
||||
|
||||
Okay, more new stuff here :) This time, items!
|
||||
|
||||
Items
|
||||
^^^^^
|
||||
|
||||
Items are the objects we use to represent what you scrape (in this case,
|
||||
links). Basically, there are two important things about items: attributes, and
|
||||
adaptors.
|
||||
|
||||
Attributes
|
||||
""""""""""
|
||||
|
||||
Attributes are nothing else but the places where you store the data you are
|
||||
extracting, which in this case are, the name of the linked website, its url,
|
||||
and a description. Now, in most cases, you'll have to do certain modifications
|
||||
to this data in order to store it (or do whatever you want to do with it), and
|
||||
this is done through the adaptors.
|
||||
|
||||
Adaptors
|
||||
""""""""
|
||||
|
||||
Adaptors are basically a list of functions that receive a value, modify it (or
|
||||
not), and then return it. In this case we used only two adaptors:
|
||||
|
||||
* ``extract``, which, as you may imagine, extracts data from the XPath nodes
|
||||
you provide, and returns it as a list.
|
||||
|
||||
* ``delist``, which joins the list that the previous adaptor returned into a
|
||||
string. This adaptor itself is a class, and this is due to the fact that you
|
||||
must specify which delimiter will join the list. That's why we put an
|
||||
instance to this adaptor in the list.
|
||||
|
||||
* ``strip``, which (as you may imagine), does the same as the python strings
|
||||
strip method. Cleans up extra spaces before and after the provided string.
|
||||
|
||||
In this case, we used the same adaptors for every attribute, because we're
|
||||
practically doing nothing to the data, just extracting it. But there might be
|
||||
situations were certain attributes are handled different than others (in fact,
|
||||
it will happen once you scrape more complicated sites with more complicated
|
||||
data).
|
||||
|
||||
The rest of the code is quite self-explanatory. The ``attribute`` method sets the
|
||||
item's attributes, and the items themselves are put into a list that we'll
|
||||
return to Scrapy's engine. One simple (although important) thing to remember
|
||||
here is that you must always return a list that contains either items,
|
||||
requests, or both, but always inside a list.
|
||||
|
||||
So, we're almost done! Let's now check the last part of the tutorial:
|
||||
:ref:`intro-tutorial4`
|
||||
|
|
@ -1,78 +0,0 @@
|
|||
.. _intro-tutorial4:
|
||||
|
||||
=================
|
||||
Finishing the job
|
||||
=================
|
||||
|
||||
Well, we've got our project, our spider, and our scraped items.
|
||||
What to do next?
|
||||
|
||||
It actually depends on what you want to do with the scraped data. In this
|
||||
case, we'll imagine that we want to save this data for storing it in a db
|
||||
later, or just to keep it there.
|
||||
|
||||
To make it simple, we'll export the scraped items to a CSV file by making use
|
||||
of a useful function that Scrapy brings: ``items_to_csv``. This simple function
|
||||
takes a file descriptor/filename, and a list of items, and writes their
|
||||
attributes to that file, in CSV format.
|
||||
|
||||
.. highlight:: python
|
||||
|
||||
Let's see how would our spider end up looking like after applying this change::
|
||||
|
||||
# -*- coding: utf8 -*-
|
||||
import re
|
||||
|
||||
from scrapy.xpath import HtmlXPathSelector
|
||||
from scrapy.link.extractors import RegexLinkExtractor
|
||||
from scrapy.contrib.spiders import CrawlSpider, Rule
|
||||
from scrapy.contrib_exp import adaptors
|
||||
from scrapy.utils.misc import items_to_csv
|
||||
from googledir.items import GoogledirItem
|
||||
|
||||
class GoogleDirectorySpider(CrawlSpider):
|
||||
domain_name = 'google.com'
|
||||
start_urls = ['http://www.google.com/dirhp']
|
||||
|
||||
rules = (
|
||||
Rule(RegexLinkExtractor(allow=('google.com/[A-Z][a-zA-Z_/]+$',),),
|
||||
'parse_category',
|
||||
follow=True,
|
||||
),
|
||||
)
|
||||
csv_file = open('scraped_items.csv', 'ab+')
|
||||
|
||||
def parse_category(self, response):
|
||||
# The selector we're going to use in order to extract data from the page
|
||||
hxs = HtmlXPathSelector(response)
|
||||
|
||||
# The path to website links in directory page
|
||||
links = hxs.x('//td[descendant::a[contains(@href, "#pagerank")]]/following-sibling::td/font')
|
||||
|
||||
# The list of functions to apply to an attribute before assigning its value
|
||||
adaptor_pipe = [adaptors.extract, adaptors.delist(''), adaptors.strip]
|
||||
adaptor_map = {
|
||||
'name': adaptor_pipe,
|
||||
'url': adaptor_pipe,
|
||||
'description': adaptor_pipe,
|
||||
}
|
||||
|
||||
for link in links:
|
||||
item = GoogledirItem()
|
||||
item.set_adaptors(adaptor_map)
|
||||
|
||||
item.attribute('name', link.x('a/text()'))
|
||||
item.attribute('url', link.x('a/@href'))
|
||||
item.attribute('description', link.x('font[2]/text()'))
|
||||
items_to_csv(self.csv_file, [item])
|
||||
yield item
|
||||
|
||||
SPIDER = GoogleDirectorySpider()
|
||||
|
||||
With this code, our spider will crawl over Google's directory, and save each
|
||||
link's name, description, and url to a file called 'scraped_items.csv'::
|
||||
|
||||
./scrapy-ctl.py crawl google.com
|
||||
|
||||
This is the end of the tutorial. If you'd like to know more about Scrapy and
|
||||
its use, please read the rest of the documentation.
|
||||
|
|
@ -23,37 +23,12 @@ relevant.
|
|||
API Stability
|
||||
=============
|
||||
|
||||
Methods or functions that start with a single ``_`` are private and should
|
||||
never be relied as stable. Besides those, the plan is to stabilize and document
|
||||
the entire API, as we approach the 1.0 release. In the meantime, you'll find
|
||||
here a list of the APIs that we consider already stable.
|
||||
Methods or functions that start with a single dash (``_``) are private and
|
||||
should never be relied as stable. Besides those, the plan is to stabilize and
|
||||
document the entire API, as we approach the 1.0 release. In the meantime,
|
||||
you'll find here a list of the APIs that we consider already stable.
|
||||
|
||||
Also, keep in mind that stable doesn't mean complete: stable APIs could grow
|
||||
new methods or functionality but the existing methods should keep working the
|
||||
same way.
|
||||
|
||||
Stable APIs
|
||||
-----------
|
||||
|
||||
The APIs listed here should keep working as documented between minor versions.
|
||||
|
||||
* :ref:`topics-items`
|
||||
* :ref:`topics-selectors`
|
||||
* :ref:`topics-item-pipeline`
|
||||
* :ref:`topics-downloader-middleware`
|
||||
* :ref:`topics-spider-middleware`
|
||||
* :ref:`topics-settings`
|
||||
|
||||
Almost stable APIs
|
||||
------------------
|
||||
|
||||
The APIs listed here may suffer minor changes between minor versions.
|
||||
|
||||
Unstable APIs
|
||||
-------------
|
||||
|
||||
These APIs may suffer major changes or be removed completely on the next minor
|
||||
version release.
|
||||
|
||||
* :ref:`topics-adaptors`
|
||||
|
||||
|
|
|
|||
|
|
@ -18,7 +18,3 @@ Proposed documentation
|
|||
|
||||
introduction
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
tutorial
|
||||
|
|
|
|||
|
|
@ -3,10 +3,10 @@
|
|||
API Reference
|
||||
=============
|
||||
|
||||
This section documents the API of Scrapy |version|. For more information see :ref:`misc-api-stability`.
|
||||
This section documents the Scrapy |version| API. For more information see :ref:`misc-api-stability`.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
:maxdepth: 2
|
||||
|
||||
spiders
|
||||
selectors
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
.. _ref-selectors:
|
||||
|
||||
=============
|
||||
Selectors API
|
||||
=============
|
||||
===================
|
||||
XPath Selectors API
|
||||
===================
|
||||
|
||||
.. module:: scrapy.xpath
|
||||
:synopsis: XPath selectors classes
|
||||
|
|
|
|||
|
|
@ -18,7 +18,9 @@ ADAPTORS_DEBUG
|
|||
|
||||
Default: ``False``
|
||||
|
||||
Enable debug mode for adaptors.
|
||||
Enable debug mode for adaptors.
|
||||
|
||||
See :ref:`topics-adaptors`.
|
||||
|
||||
.. setting:: BOT_NAME
|
||||
|
||||
|
|
|
|||
|
Before Width: | Height: | Size: 43 KiB After Width: | Height: | Size: 43 KiB |
|
Before Width: | Height: | Size: 68 KiB After Width: | Height: | Size: 68 KiB |
|
Before Width: | Height: | Size: 88 KiB After Width: | Height: | Size: 88 KiB |
|
|
@ -1,8 +1,8 @@
|
|||
.. _topics-adaptors:
|
||||
|
||||
========
|
||||
Adaptors
|
||||
========
|
||||
=======================
|
||||
Adaptors (experimental)
|
||||
=======================
|
||||
|
||||
.. warning::
|
||||
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
.. _topics-architecture:
|
||||
|
||||
============
|
||||
Architecture
|
||||
============
|
||||
=====================
|
||||
Architecture overview
|
||||
=====================
|
||||
|
||||
This document describes the architecture of Scrapy and how their components
|
||||
interact.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,154 @@
|
|||
.. _topics-firebug:
|
||||
|
||||
==========================
|
||||
Using Firebug for scraping
|
||||
==========================
|
||||
|
||||
Introduction
|
||||
============
|
||||
|
||||
This document explains how to use `Firebug`_ (a Firefox add-on) to make the
|
||||
scraping process easier and more fun. For other useful Firefox add-ons see
|
||||
:ref:`topics-firefox`.
|
||||
|
||||
In this example, we'll show how to use `Firebug`_ to scrape data from the
|
||||
`Google Directory`_, which contains the same data as the `Open Directory
|
||||
Project`_ used in the :ref:`tutorial <intro-tutorial>` but with a different
|
||||
face.
|
||||
|
||||
.. _Firebug: http://getfirebug.com
|
||||
.. _Google Directory: http://www.google.com/dirhp
|
||||
.. _Open Directory Project: http://www.dmoz.org
|
||||
|
||||
Firebug comes with a very useful feature called `Inspect Element`_ which allows
|
||||
you to inspect the HTML code of the different page elements just by hovering
|
||||
your mouse over them. Otherwise you would have to search for the tags manually
|
||||
through the HTML body which can be a very tedious task.
|
||||
|
||||
.. _Inspect Element: http://www.youtube.com/watch?v=-pT_pDe54aA
|
||||
|
||||
In the following screenshot you can see the `Inspect Element`_ tool in action.
|
||||
|
||||
.. image:: _images/firebug1.png
|
||||
:width: 913
|
||||
:height: 600
|
||||
:alt: Inspecting elements with Firebug
|
||||
|
||||
At first sight, we can see that the directory is divided in categories, which
|
||||
are also divided in subcategories.
|
||||
|
||||
However, it seems as if there are more subcategories than the ones being shown
|
||||
in this page, so we'll keep looking:
|
||||
|
||||
.. image:: _images/firebug2.png
|
||||
:width: 819
|
||||
:height: 629
|
||||
:alt: Inspecting elements with Firebug
|
||||
|
||||
As expected the subcategories contain links to other subcategories, and also
|
||||
links to actual websites, which is the purpose of the directory.
|
||||
|
||||
Getting links to follow
|
||||
=======================
|
||||
|
||||
By looking at the category URLs we can see they share a pattern:
|
||||
|
||||
http://www.google.com/Category/Subcategory/Another_Subcategory
|
||||
|
||||
Once we know that, we are able to construct a regular expression to follow
|
||||
those links. For example, the following one::
|
||||
|
||||
google.com/[A-Z][a-zA-Z_/]+$
|
||||
|
||||
So, based on that regular expression we can create the first crawling rule::
|
||||
|
||||
Rule(RegexLinkExtractor(allow='google.com/[A-Z][a-zA-Z_/]+$', ),
|
||||
'parse_category',
|
||||
follow=True,
|
||||
),
|
||||
|
||||
The :class:`~scrapy.contrib.spiders.Rule` object instructs
|
||||
:class:`~scrapy.contrib.spiders.CrawlSpider` based spiders how to follow the
|
||||
category links. ``parse_category`` will be a method of the spider which will
|
||||
process and extract data from those pages.
|
||||
|
||||
This is how the spider would look so far::
|
||||
|
||||
from scrapy.link.extractors import RegexLinkExtractor
|
||||
from scrapy.contrib.spiders import CrawlSpider, Rule
|
||||
|
||||
class GoogleDirectorySpider(CrawlSpider):
|
||||
domain_name = 'google.com'
|
||||
start_urls = ['http://www.google.com/dirhp']
|
||||
|
||||
rules = (
|
||||
Rule(RegexLinkExtractor(allow='google.com/[A-Z][a-zA-Z_/]+$'),
|
||||
'parse_category', follow=True,
|
||||
),
|
||||
)
|
||||
|
||||
def parse_category(self, response):
|
||||
# write the category page data extraction code here
|
||||
pass
|
||||
|
||||
SPIDER = GoogleDirectorySpider()
|
||||
|
||||
|
||||
Extracting the data
|
||||
===================
|
||||
|
||||
Now we're gonna write the code to extract data from those pages.
|
||||
|
||||
With the help of Firebug, we'll take a look at some page containing links to
|
||||
websites (say http://www.google.com/Top/Arts/Awards/) and find out how we can
|
||||
extract those links using :ref:`XPath selectors <topics-selectors>`. We'll also
|
||||
use the :ref:`Scrapy shell <topics-shell>` to test those XPath's and make sure
|
||||
they work as we expect.
|
||||
|
||||
.. image:: _images/firebug3.png
|
||||
:width: 965
|
||||
:height: 751
|
||||
:alt: Inspecting elements with Firebug
|
||||
|
||||
As you can see, the page markup is not very descriptive: the elements don't
|
||||
contain ``id``, ``class`` or any attribute that clearly identifies them, so
|
||||
we''ll use the ranking bars as a reference point to select the data to extract
|
||||
when we construct our XPaths.
|
||||
|
||||
After using FireBug, we can see that each link is inside a ``td`` tag, which is
|
||||
itself inside a ``tr`` tag that also contains the link's ranking bar (in
|
||||
another ``td``).
|
||||
|
||||
So we can select the ranking bar, then find its parent (the ``tr``), and then
|
||||
finally, the link's ``td`` (which contains the data we want to scrape).
|
||||
|
||||
This results in the following XPath::
|
||||
|
||||
//td[descendant::a[contains(@href, "#pagerank")]]/following-sibling::td//a
|
||||
|
||||
It's important to use the :ref:`Scrapy shell <topics-shell>` to test these
|
||||
complex XPath expressions and make sure they work as expected.
|
||||
|
||||
Basically, that expression will look for the ranking bar's ``td`` element, and
|
||||
then select any ``td`` element who has a descendant ``a`` element whose
|
||||
``href`` attribute contains the string ``#pagerank``"
|
||||
|
||||
Of course, this is not the only XPath, and maybe not the simpler one to select
|
||||
that data. Another approach could be, for example, to find any ``font`` tags
|
||||
that have that grey colour of the links,
|
||||
|
||||
Finally, we can write our ``parse_category()`` method::
|
||||
|
||||
def parse_category(self, response):
|
||||
hxs = HtmlXPathSelector(response)
|
||||
|
||||
# The path to website links in directory page
|
||||
links = hxs.x('//td[descendant::a[contains(@href, "#pagerank")]]/following-sibling::td/font')
|
||||
|
||||
for link in links:
|
||||
item = ScrapedItem()
|
||||
item.name = link.x('a/text()').extract()
|
||||
item.url = link.x('a/@href').extract()
|
||||
item.description = link.x('font[2]/text()').extract()
|
||||
yield item
|
||||
|
||||
|
|
@ -0,0 +1,52 @@
|
|||
.. _topics-firefox:
|
||||
|
||||
===================================
|
||||
Useful Firefox add-ons for scraping
|
||||
===================================
|
||||
|
||||
Here is a list of useful Firefox add-ons you can use to make the scraping
|
||||
process easier, faster and more comfortable.
|
||||
|
||||
Firebug
|
||||
=======
|
||||
|
||||
`Firebug`_ is a widely known tool among web developers and it's also very
|
||||
useful for scraping. In particular, its `Inspect Element`_ feature comes very
|
||||
handy when you need to construct the XPaths for extracting data because it
|
||||
allows you to view the HTML code of each page element while moving your mouse
|
||||
over them.
|
||||
|
||||
See :ref:`topics-firebug` for a detailed guide on how to use Firebug with
|
||||
Scrapy.
|
||||
|
||||
XPather
|
||||
=======
|
||||
|
||||
`XPather`_ allows you to test XPath expressions directly on the pages.
|
||||
|
||||
XPath Checker
|
||||
=============
|
||||
|
||||
`XPath Checker`_ is another Firefox add-on for testing XPaths on your pages.
|
||||
|
||||
Tamper Data
|
||||
===========
|
||||
|
||||
`Tamper Data`_ is a Firefox add-on which allows you to view and modify the HTTP
|
||||
request headers sent by Firefox. Firebug also allows to view HTTP headers, but
|
||||
not to modify them.
|
||||
|
||||
Firecookie
|
||||
==========
|
||||
|
||||
`Firecookie`_ makes it easier to view and manage cookies. You can use this
|
||||
extension to create a new cookie, delete existing cookies, see list of cookies
|
||||
for current site, manage cookies permissions and a lot more.
|
||||
|
||||
.. _Firebug: http://getfirebug.com
|
||||
.. _Inspect Element: http://www.youtube.com/watch?v=-pT_pDe54aA
|
||||
.. _XPather: https://addons.mozilla.org/firefox/addon/1192
|
||||
.. _XPath Checker: https://addons.mozilla.org/firefox/addon/1095
|
||||
.. _Tamper Data: http://addons.mozilla.org/firefox/addon/966
|
||||
.. _Firecookie: https://addons.mozilla.org/firefox/addon/6683
|
||||
|
||||
|
|
@ -3,7 +3,7 @@
|
|||
Using Scrapy
|
||||
============
|
||||
|
||||
This section introduces all key concepts of Scrapy.
|
||||
This section describes all key concepts of Scrapy.
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
|
@ -14,7 +14,6 @@ This section introduces all key concepts of Scrapy.
|
|||
selectors
|
||||
link-extractors
|
||||
items
|
||||
adaptors
|
||||
item-pipeline
|
||||
downloader-middleware
|
||||
spider-middleware
|
||||
|
|
@ -23,3 +22,5 @@ This section introduces all key concepts of Scrapy.
|
|||
stats
|
||||
webconsole
|
||||
robotstxt
|
||||
firebug
|
||||
firefox
|
||||
|
|
|
|||
|
|
@ -1,14 +1,14 @@
|
|||
.. _topics-selectors:
|
||||
|
||||
=========
|
||||
Selectors
|
||||
=========
|
||||
===============
|
||||
XPath Selectors
|
||||
===============
|
||||
|
||||
Introduction
|
||||
------------
|
||||
|
||||
When you're scraping web pages, the most common task you need to perform is
|
||||
extract data from the HTML source. There are several libraries available to
|
||||
to extract data from the HTML source. There are several libraries available to
|
||||
achieve this:
|
||||
|
||||
* `BeautifulSoup`_ is a very popular screen scraping library among Python
|
||||
|
|
@ -20,12 +20,12 @@ achieve this:
|
|||
API based on `ElementTree`_ (which is not part of the Python standard
|
||||
library).
|
||||
|
||||
Scrapy comes with its own mechanism for mechanism for extracting data. They're
|
||||
called selectors, because they "select" certain parts of the HTML document
|
||||
specified by XPaths.
|
||||
Scrapy comes with its own mechanism for extracting data. They're called XPath
|
||||
selectors (or just "selectors", for short) because they "select" certain parts
|
||||
of the HTML document specified by `XPath`_ expressions.
|
||||
|
||||
`XPath`_ is a language for selecting nodes in XML documents, and it's also
|
||||
applicable for HTML documents.
|
||||
`XPath`_ is a language for selecting nodes in XML documents, which can be used
|
||||
to with HTML.
|
||||
|
||||
Both `lxml`_ and Scrapy Selectors are built over the `libxml2`_ library, which
|
||||
means they're very similar in speed and parsing accuracy.
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
.. _topics-shell:
|
||||
|
||||
=====
|
||||
Shell
|
||||
=====
|
||||
================
|
||||
The Scrapy shell
|
||||
================
|
||||
|
||||
The Scrapy shell is an interactive shell where you can try and debug your
|
||||
scraping code very quickly, without having to run the spider. It's meant to be
|
||||
|
|
@ -109,7 +109,7 @@ shell works.
|
|||
|
||||
::
|
||||
|
||||
scrapy-ctl.py shell http://scrapy.org
|
||||
python scrapy-ctl.py shell http://scrapy.org
|
||||
|
||||
2009-04-02 16:56:22-0300 [-] Log opened.
|
||||
Welcome to Scrapy shell!
|
||||
|
|
@ -125,8 +125,8 @@ shell works.
|
|||
item: <class 'myproject.models.Item'>
|
||||
response: <http://scrapy.org>
|
||||
Available commands:
|
||||
get <url>: Fetches an url and updates all variables.
|
||||
scrapehelp: Prints this help.
|
||||
get [url]: Fetch a new URL or re-fetch current Request
|
||||
shelp: Prints this help.
|
||||
------------------------------------------------------------
|
||||
Python 2.5.2 (r252:60911, Oct 5 2008, 19:29:17)
|
||||
Type "copyright", "credits" or "license" for more information.
|
||||
|
|
|
|||
Loading…
Reference in New Issue