mirror of https://github.com/scrapy/scrapy.git
added complete spiders topic and reference (in one file) using an autodoc and manual doc mix
--HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40831
This commit is contained in:
Ismael Carnales
parent
3fb945be7b
commit
5ca4728805
|
|
@ -16,3 +16,7 @@ Proposed documentation
|
|||
:maxdepth: 1
|
||||
|
||||
tutorial
|
||||
|
||||
.. toctree::
|
||||
|
||||
spiders
|
||||
|
|
|
|||
|
|
@ -4,8 +4,8 @@ Introduction
|
|||
|
||||
.. architecture:
|
||||
|
||||
Architecture
|
||||
============
|
||||
Overview
|
||||
========
|
||||
|
||||
.. image:: _images/scrapy_architecture.png
|
||||
:width: 700
|
||||
|
|
@ -15,7 +15,7 @@ Architecture
|
|||
.. _items:
|
||||
|
||||
Items
|
||||
=====
|
||||
-----
|
||||
|
||||
In Scrapy, Items are the placeholder to use for the scraped data. They are
|
||||
represented by a :class:`~scrapy.item.ScrapedItem` object, or any subclass
|
||||
|
|
@ -24,7 +24,7 @@ instance, and store the information in instance attributes.
|
|||
.. _request-response:
|
||||
|
||||
Requests and Responses
|
||||
======================
|
||||
----------------------
|
||||
|
||||
Scrapy uses :class:`~scrapy.http.Request` and :class:`~scrapy.http.Response`
|
||||
objects for crawling web sites.
|
||||
|
|
@ -36,10 +36,10 @@ Downloader, which actually executes the request and returns a
|
|||
:class:`~scrapy.http.Response` object to the :class:`Request's callback
|
||||
function <scrapy.http.Request>`.
|
||||
|
||||
.. _spiders:
|
||||
.. _overview-spiders:
|
||||
|
||||
Spiders
|
||||
=======
|
||||
-------
|
||||
|
||||
Spiders are user written classes which define how a certain site (or domain)
|
||||
will be scraped; including how to crawl the site and how to scrape :ref:`Items
|
||||
|
|
@ -48,47 +48,10 @@ will be scraped; including how to crawl the site and how to scrape :ref:`Items
|
|||
All Spiders must be descendant of :class:`~scrapy.spider.BaseSpider` or any
|
||||
subclass of it, in :ref:`ref-spiders` you can see a list of available Spiders
|
||||
in Scrapy.
|
||||
|
||||
Scraping cycle
|
||||
--------------
|
||||
|
||||
1. *Generating Requests*:
|
||||
|
||||
The first step is to generate the initial :ref:`Requests
|
||||
<request-response>` to crawl the first URLs, and specify a callback
|
||||
function to be called with the :ref:`Response <request-response>`
|
||||
downloaded from those :ref:`Requests <request-response>`.
|
||||
|
||||
The first :ref:`Requests <request-response>`. to perform are obtained by
|
||||
calling the :meth:`~scrapy.spider.BaseSpider.start_requests` method which
|
||||
(by default) generates :class:`~scrapy.http.Request` for the URLs specified
|
||||
in the :attr:`BaseSpider.start_urls` and the
|
||||
:meth:`~scrapy.spider.BaseSpider.parse` method as callback function for the
|
||||
:ref:`Requests <request-response>`..
|
||||
|
||||
2. *Parsing Responses*:
|
||||
|
||||
In callback functions you parse the :ref:`Response <request-response>`
|
||||
contents and return an iterable object containing :ref:`Items <items>`,
|
||||
:ref:`Requests <request-response>`, or both.
|
||||
|
||||
Typically you do the parsing by using :ref:`selectors`, but you could also
|
||||
use BeautifuSoup, lxml or the mechanism of your choice.
|
||||
|
||||
3. *The final step*:
|
||||
|
||||
Returned :ref:`Requests <request-response>` (if any) will be downloaded by
|
||||
Scrapy and their :ref:`Responses <request-response>` handled to the
|
||||
specified callback, wich could be the same than the one specified in the
|
||||
first step.
|
||||
|
||||
Returned :ref:`Items <items>`. (if any) will be directed to the :ref:`Item
|
||||
Pipeline <item-pipeline>`.
|
||||
|
||||
.. _selectors:
|
||||
|
||||
Selectors
|
||||
=========
|
||||
---------
|
||||
|
||||
Selectors are the recommended tool to extract information from documents. They
|
||||
retrieve information from the :ref:`Response <request-response>` body using
|
||||
|
|
@ -108,22 +71,16 @@ node, or the entire document.
|
|||
.. _item-pipeline:
|
||||
|
||||
Item Pipeline
|
||||
=============
|
||||
-------------
|
||||
|
||||
After an :ref:`Item <items>` has been scraped by a :ref:`Spider <spiders>`, it
|
||||
is sent to the Item Pipeline which allows us to perform some actions over the
|
||||
:ref:`scrapped Items <items>`.
|
||||
|
||||
The Item Pipeline is a list of user written Python classes that define the
|
||||
:meth:`process_item` method, which is called sequentially for every element.
|
||||
The Item Pipeline is a list of user written Python classes that implement a
|
||||
specific method , which is called sequentially for every element of the
|
||||
Pipeline.
|
||||
|
||||
The :meth:`process_item` must return the Item object on a successful action,
|
||||
or raise a :exception:`DropItem` exception (ex: failing a validation test).
|
||||
Dropped :ref:`Items <items>` are no longer processed by further pipeline
|
||||
components.
|
||||
|
||||
Typical uses of the Item Pipeline include:
|
||||
|
||||
* Clean the HTML in Item attributes
|
||||
* Validate the Item
|
||||
* Store the Item
|
||||
Each element receives the Scraped Item, do an action upon it (like validating,
|
||||
checking for duplicates, store the item), and then decide if the Item
|
||||
continues trough the Pipeline or the item is dropped.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,229 @@
|
|||
.. _spiders:
|
||||
|
||||
Spiders
|
||||
=======
|
||||
|
||||
Spiders are user written classes which define how a certain site (or domain)
|
||||
will be scraped; including how to crawl the site and how to scrape :ref:`Items
|
||||
<items>` from their pages.
|
||||
|
||||
All Spiders must be descendant of :class:`~scrapy.spider.BaseSpider` or any
|
||||
subclass of it, below you can see a list of available Spiders in Scrapy.
|
||||
|
||||
.. _spiders-ref:
|
||||
|
||||
Available Spiders
|
||||
=================
|
||||
|
||||
.. module:: scrapy.spider
|
||||
|
||||
BaseSpider
|
||||
----------
|
||||
|
||||
.. autoclass:: BaseSpider(object)
|
||||
:members:
|
||||
|
||||
.. attribute:: domain_name
|
||||
|
||||
A string which defines the domain name for this spider, which will also
|
||||
be the unique identifier for this spider (which means you can't have two
|
||||
spider with the same :attr:`domain_name`). This is the most important
|
||||
spider attribute and it's required, and it's the name by which Scrapy
|
||||
will known the spider.
|
||||
|
||||
.. attribute:: extra_domain_names
|
||||
|
||||
An optional list of strings containing additional domains that this
|
||||
spider is allowed to crawl. Requests for URLs not belonging to the domain
|
||||
name specified in :attr:`domain_name` or this list won't be followed.
|
||||
|
||||
.. attribute:: start_urls
|
||||
|
||||
Is a list of URLs where the spider will begin to crawl from, when no
|
||||
particular URLs are specified. So, the first pages downloaded will be
|
||||
those listed here. The subsequent URLs will be generated successively
|
||||
from data contained in the start URLs.
|
||||
|
||||
BaseSpider example
|
||||
^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Let's see an example::
|
||||
|
||||
from scrapy import log # This module is useful for printing out debug information
|
||||
from scrapy.spider import BaseSpider
|
||||
|
||||
class MySpider(BaseSpider):
|
||||
domain_name = 'http://www.example.com'
|
||||
start_urls = [
|
||||
'http://www.example.com/1.html',
|
||||
'http://www.example.com/2.html',
|
||||
'http://www.example.com/3.html',
|
||||
]
|
||||
|
||||
def parse(self, response):
|
||||
log.msg('Hey! A response from %s has just arrived!' % response.url)
|
||||
return []
|
||||
|
||||
SPIDER = MySpider()
|
||||
|
||||
.. module:: scrapy.contrib.spiders
|
||||
|
||||
CrawlSpider
|
||||
-----------
|
||||
|
||||
.. autoclass:: CrawlSpider(BaseSpider)
|
||||
:members:
|
||||
|
||||
.. attribute:: CrawlSpider.rules
|
||||
|
||||
Which is a list of one (or more) :class:`Rule` objects. Each
|
||||
:class:`Rule` defines a certain behaviour for crawling the site. Rules
|
||||
objects are described below.
|
||||
|
||||
Crawling rules
|
||||
^^^^^^^^^^^^^^
|
||||
|
||||
.. autoclass:: Rule(link_extractor, callback=None, cb_kwargs=None, follow=None, process_links=None)
|
||||
|
||||
CrawlSpider example
|
||||
^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Let's now take a look at an example CrawlSpider with Rules::
|
||||
|
||||
from scrapy import log
|
||||
from scrapy.contrib.spiders import CrawlSpider, Rule
|
||||
from scrapy.link.extractors import RegexLinkExtractor
|
||||
from scrapy.xpath.selector import HtmlXPathSelector
|
||||
from scrapy.item import ScrapedItem
|
||||
|
||||
class MySpider(CrawlSpider):
|
||||
domain_name = 'example.com'
|
||||
start_urls = ['http://www.example.com']
|
||||
|
||||
rules = (
|
||||
# Extract links matching 'category.php' (but not matching 'subsection.php')
|
||||
# and follow links from them (since no callback means follow=True by default).
|
||||
Rule(RegexLinkExtractor(allow=('category\.php', ), deny=('subsection\,php', ))),
|
||||
|
||||
# Extract links matching 'item.php' and parse them with the spider's method parse_item
|
||||
Rule(RegexLinkExtractor(allow=('item\.php', )), callback='parse_item'),
|
||||
)
|
||||
|
||||
def parse_item(self, response):
|
||||
log.msg('Hi, this is an item page! %s' % response.url)
|
||||
|
||||
hxs = HtmlXPathSelector(response)
|
||||
item = ScrapedItem()
|
||||
item.attribute('id', hxs.x('//td[@id="item_id"]/text()').re(r'ID: (\d+)'))
|
||||
item.attribute('name', hxs.x('//td[@id="item_name"]/text()'))
|
||||
item.attributE('description', hxs.x('//td[@id="item_description"]/text()'))
|
||||
return [item]
|
||||
|
||||
SPIDER = MySpider()
|
||||
|
||||
This spider would start crawling example.com's home page, collecting category
|
||||
links, and item links, parsing the latter with the *parse_item* method. For
|
||||
each item response, some data will be extracted from the HTML using XPath, and
|
||||
a ScrapedItem will be filled with it.
|
||||
|
||||
XMLFeedSpider
|
||||
-------------
|
||||
|
||||
.. autoclass:: XMLFeedSpider(BaseSpider)
|
||||
:members:
|
||||
|
||||
.. attribute:: iterator
|
||||
|
||||
A string which defines the iterator to use. It can be either:
|
||||
|
||||
* 'iternodes': a fast iterator based on regular expressions
|
||||
|
||||
* 'html': an iterator which uses HtmlXPathSelector. Keep in mind this
|
||||
uses DOM parsing and must load all DOM in memory which could be a
|
||||
problem for big feeds
|
||||
|
||||
* 'xml': an iterator which uses XmlXPathSelector. Keep in mind this uses
|
||||
DOM parsing and must load all DOM in memory which could be a problem
|
||||
for big feeds
|
||||
|
||||
It defaults to: 'iternodes'.
|
||||
|
||||
.. attribute:: itertag
|
||||
|
||||
A string with the name of the node (or element) to iterate in.
|
||||
|
||||
|
||||
XMLFeedSpider example
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
These spiders are pretty easy to use, let's have at one example::
|
||||
|
||||
from scrapy import log
|
||||
from scrapy.contrib.spiders import XMLFeedSpider
|
||||
from scrapy.item import ScrapedItem
|
||||
|
||||
class MySpider(XMLFeedSpider):
|
||||
domain_name = 'example.com'
|
||||
start_urls = ['http://www.example.com/feed.xml']
|
||||
iterator = 'iternodes' # This is actually unnecesary, since it's the default value
|
||||
itertag = 'item'
|
||||
|
||||
def parse_nodes(self, response, node):
|
||||
log.msg('Hi, this is a <%s> node!: %s' % (self.itertag, ''.join(node.extract())))
|
||||
|
||||
item = ScrapedItem()
|
||||
item.attribute('id', node.x('@id'))
|
||||
item.attribute('name', node.x('name'))
|
||||
item.attribute('description', node.x('description'))
|
||||
return item
|
||||
|
||||
SPIDER = MySpider()
|
||||
|
||||
Basically what we did up there was creating a spider that downloads a feed from
|
||||
the given ``start_urls``, and then iterates through each of its ``item`` tags,
|
||||
prints them out, and stores some random data in ScrapedItems.
|
||||
|
||||
CSVFeedSpider
|
||||
-------------
|
||||
|
||||
.. warning:: The API of the CSVFeedSpider is not yet stable. Use with caution.
|
||||
|
||||
.. autoclass:: CSVFeedSpider(BaseSpider)
|
||||
:members:
|
||||
|
||||
.. attribute:: CSVFeedSpider.delimiter
|
||||
|
||||
A string with the separator character for each field in the CSV file
|
||||
Defaults to ``','`` (comma).
|
||||
|
||||
.. attribute:: CSVFeedSpider.headers
|
||||
|
||||
A list of the rows contained in the file CSV feed which will be used for
|
||||
extracting fields from it.
|
||||
|
||||
CSVFeedSpider example
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Let's see an example similar to the previous one, but using CSVFeedSpider::
|
||||
|
||||
from scrapy import log
|
||||
from scrapy.contrib.spiders import CSVFeedSpider
|
||||
from scrapy.item import ScrapedItem
|
||||
|
||||
class MySpider(CSVFeedSpider):
|
||||
domain_name = 'example.com'
|
||||
start_urls = ['http://www.example.com/feed.csv']
|
||||
delimiter = ';'
|
||||
headers = ['id', 'name', 'description']
|
||||
|
||||
def parse_rows(self, response, row):
|
||||
log.msg('Hi, this is a row!: %r' % row)
|
||||
|
||||
item = ScrapedItem()
|
||||
item.attribute('id', row['id'])
|
||||
item.attribute('name', row['name'])
|
||||
item.attribute('description', row['description'])
|
||||
return item
|
||||
|
||||
SPIDER = MySpider()
|
||||
|
||||
Loading…
Reference in New Issue