mirror of https://github.com/scrapy/scrapy.git
splitted spiders doc from link extractor docs, moved the corresponding parts to ref and topics
--HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%40812
This commit is contained in:
Pablo Hoffman
parent
483ef3ba7f
commit
c672223091
|
|
@ -8,6 +8,7 @@ This section documents the API of Scrapy |version|. For more information see :re
|
|||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
spiders
|
||||
exceptions
|
||||
request-response
|
||||
extension-manager
|
||||
|
|
@ -17,3 +18,4 @@ This section documents the API of Scrapy |version|. For more information see :re
|
|||
signals
|
||||
logging
|
||||
email
|
||||
link-extractors
|
||||
|
|
|
|||
|
|
@ -0,0 +1,60 @@
|
|||
.. _ref-link-extractors:
|
||||
|
||||
=========================
|
||||
Available Link Extractors
|
||||
=========================
|
||||
|
||||
LinkExtractor
|
||||
=============
|
||||
|
||||
.. class:: LinkExtractor(tag="a", href="href", unique=False)
|
||||
|
||||
This is the most basic Link Extractor which extracts links from a response with
|
||||
by looking at the given attributes inside the given tags.
|
||||
|
||||
``tag`` is either a string (with the name of a tag) or a function that receives
|
||||
a tag name and returns True if links should be extracted from it, or False if
|
||||
they shouldn't. Defaults to 'a'.
|
||||
|
||||
``attr`` is either a string (with the name of an tag attribute), or a function
|
||||
that receives a an attribute name and returns True if links should be extracted from it, or False if the shouldn't.
|
||||
|
||||
``unique`` is a boolean that specifies if a duplicate filtering should be
|
||||
applied to links extracted.
|
||||
|
||||
RegexLinkExtractor
|
||||
==================
|
||||
|
||||
.. class:: RegexLinkExtractor(allow=(), deny=(), allow_domains=(), deny_domains=(), restrict_xpaths(), tags=('a', 'area'), attrs=('href'), canonicalize=True, unique=True)
|
||||
|
||||
This Link Extractor extracts links from a response by applying several filters
|
||||
that you can specify, including regular expressions that match (or don't match)
|
||||
the extracted links. These parameters are configured when instantiating the
|
||||
RegexLinkExtractor object.
|
||||
|
||||
``allow`` is a list of regular expressions that the (absolute) urls must match
|
||||
in order to be extracted. deny: A list of regular expressions that makes any
|
||||
url matching them be ignored. allow_domains: A list of domains from which to
|
||||
extract urls. deny_domains: A list of domains to not extract urls from.
|
||||
restrict_xpaths: Only extract links from the areas inside the provided xpaths
|
||||
(in a list). tags: List of tags to extract links from. Defaults to ('a',
|
||||
'area'). attrs: List of attributes to extract links from. Defaults to ('href',
|
||||
) canonicalize: Canonicalize each extracted url (using
|
||||
scrapy.utils.url.canonicalize_url). Defaults to True.
|
||||
|
||||
``allow_domains`` is a list of string containing domains which will be
|
||||
considered for extracting the links
|
||||
|
||||
``deny_domains`` is a list of strings containing domains which which won't be
|
||||
considered for extracting the links
|
||||
|
||||
``restrict_xpaths`` is a list of string with XPath's. If specified, links will
|
||||
only be looked inside the sections of the pages specified by those XPaths.
|
||||
|
||||
``tags`` is an iterable with the name of the tags where links should be extracted from
|
||||
|
||||
``attrs`` is an interable with the name of the attributes where links should be extracted from
|
||||
|
||||
``unique`` is a boolean that specifies if a duplicate filtering should be
|
||||
applied to links extracted.
|
||||
|
||||
|
|
@ -0,0 +1,340 @@
|
|||
.. _ref-spiders:
|
||||
|
||||
=================
|
||||
Available Spiders
|
||||
=================
|
||||
|
||||
.. module:: scrapy.item
|
||||
|
||||
BaseSpider
|
||||
==========
|
||||
|
||||
.. class:: BaseSpider()
|
||||
|
||||
This is the simplest spider, and the one from which every other spider
|
||||
must inherit from (either the ones that come bundled with Scrapy, or the ones
|
||||
that you write yourself). It doesn't provide any special functionality. It just
|
||||
requests the given ``start_urls``/``start_requests``, and calls the spider's
|
||||
method ``parse`` for each of the resulting responses.
|
||||
|
||||
.. attribute:: BaseSpider.domain_name
|
||||
|
||||
A string which defines the domain name for this spider, which will also be
|
||||
the unique identifier for this spider (which means you can't have two
|
||||
spider with the same ``domain_name``). This is the most important spider
|
||||
attribute and it's required, and it's the name by which Scrapy will known
|
||||
the spider.
|
||||
|
||||
.. attribute:: BaseSpider.extra_domain_names
|
||||
|
||||
An optional list of strings containing additional domains that this spider
|
||||
is allowed to crawl. Requests for URLs not belonging to the domain name
|
||||
specified in :attr:`Spider.domain_name` or this list won't be followed.
|
||||
|
||||
.. attribute:: BaseSpider.start_urls
|
||||
|
||||
Is a list of URLs where the spider will begin to crawl from, when no
|
||||
particular URLs are specified. So, the first pages downloaded will be those
|
||||
listed here. The subsequent URLs will be generated successively from data
|
||||
contained in the start URLs.
|
||||
|
||||
.. method:: BaseSpider.start_requests(urls=None)
|
||||
|
||||
A method that receives a list of URLs to scrape (for that spider) and
|
||||
returns a list of Requests for those urls.
|
||||
|
||||
If urls is `None` it will use the :attr:`BaseSpider.start_urls` attribute.
|
||||
|
||||
Unless overriden, the Requests returned by this method will use the
|
||||
:meth:`BaseSpider.parse` method as their callback function.
|
||||
|
||||
This is also the first method called by Scrapy when it opens a spider for
|
||||
scraping, so you if you want to change the Requests used to start scraping
|
||||
a domain, this is the method to override. For example, if you need to start
|
||||
by login in using a POST request, you could do::
|
||||
|
||||
def start_requests(self):
|
||||
return [FormRequest("http://www.example.com/login",
|
||||
formdata={'user': 'john', 'pass': 'secret'},
|
||||
callback=self.logged_in)]
|
||||
|
||||
def logged_in(self, response):
|
||||
# here you would extract links to follow and return Requests for
|
||||
# each of them, perhaps with another callback
|
||||
pass
|
||||
|
||||
.. method:: BaseSpider.parse(response)
|
||||
|
||||
This is the default callback used by the :meth:`start_requests` method, and
|
||||
will be used to parse the first pages crawled by the spider.
|
||||
|
||||
The ``parse`` method is in charge of processing the response and returning
|
||||
scraped data and/or more URLs to follow, because of this, the method must
|
||||
always return a list or at least an empty one. Other Requests callbacks
|
||||
have the same requirements as the BaseSpider class.
|
||||
|
||||
BaseSpider example
|
||||
------------------
|
||||
|
||||
Let's see an example::
|
||||
|
||||
from scrapy import log # This module is useful for printing out debug information
|
||||
from scrapy.spider import BaseSpider
|
||||
|
||||
class MySpider(BaseSpider):
|
||||
domain_name = 'http://www.example.com'
|
||||
start_urls = [
|
||||
'http://www.example.com/1.html',
|
||||
'http://www.example.com/2.html',
|
||||
'http://www.example.com/3.html',
|
||||
]
|
||||
|
||||
def parse(self, response):
|
||||
log.msg('Hey! A response from %s has just arrived!' % response.url)
|
||||
return []
|
||||
|
||||
SPIDER = MySpider()
|
||||
|
||||
.. module:: scrapy.contrib.spiders
|
||||
|
||||
CrawlSpider
|
||||
===========
|
||||
|
||||
.. class:: CrawlSpider
|
||||
|
||||
This is the most commonly used spider, and it's the one preferred for crawling
|
||||
standard web sites (ie. HTML pages), extracts links from there (given certain
|
||||
extraction rules), and scrapes items from those pages.
|
||||
|
||||
This spider is a bit more complicated than the previous one, because it
|
||||
introduces a few new concepts, but you'll probably find it useful.
|
||||
|
||||
Apart from the attributes inherited from BaseSpider (that you must
|
||||
specify), this class supports a new attribute:
|
||||
|
||||
.. attribute:: CrawlSpider.rules
|
||||
|
||||
Which is a list of one (or more) :class:`Rule` objects. Each :class:`Rule`
|
||||
defines a certain behaviour for crawling the site. Rules objects are
|
||||
described below .
|
||||
|
||||
Crawling rules
|
||||
--------------
|
||||
|
||||
.. class:: Rule(link_extractor, callback=None, cb_kwargs=None, follow=None, process_links=None)
|
||||
|
||||
``link_extractor`` is a :ref:`Link Extractor <topics-link-extractors>` object which
|
||||
defines how links will be extracted from each crawled page.
|
||||
|
||||
``callback`` is a callable or a string (in which case a method from the spider
|
||||
object with that name will be used) to be called for each link extracted with
|
||||
the specified link_extractor. This callback receives a response as its first
|
||||
argument and must return a list containing either ScrapedItems and Requests (or
|
||||
any subclass of them).
|
||||
|
||||
``cb_kwargs`` is a dict containing the keyword arguments to be passed to the
|
||||
callback function
|
||||
|
||||
``follow`` is a boolean which specified if links should be followed from each
|
||||
response extracted with this rule. If ``callback`` is None ``follow`` defaults
|
||||
to ``True``, otherwise it default to ``False``.
|
||||
|
||||
``process_links`` is a callable, or a string (in which case a method from the
|
||||
spider object with that name will be used) which will be called for each list
|
||||
of links extracted from each response using the specified ``link_extractor``.
|
||||
This is mainly used for filtering purposes.
|
||||
|
||||
|
||||
CrawlSpider example
|
||||
-------------------
|
||||
|
||||
Let's now take a look at an example CrawlSpider with rules::
|
||||
|
||||
from scrapy import log
|
||||
from scrapy.contrib.spiders import CrawlSpider, Rule
|
||||
from scrapy.link.extractors import RegexLinkExtractor
|
||||
from scrapy.xpath.selector import HtmlXPathSelector
|
||||
from scrapy.item import ScrapedItem
|
||||
|
||||
class MySpider(CrawlSpider):
|
||||
domain_name = 'example.com'
|
||||
start_urls = ['http://www.example.com']
|
||||
|
||||
rules = (
|
||||
# Extract links matching 'category.php' (but not matching 'subsection.php') and follow links from them (since no callback means follow=True by default).
|
||||
Rule(RegexLinkExtractor(allow=('category\.php', ), deny=('subsection\,php', ))),
|
||||
|
||||
# Extract links matching 'item.php' and parse them with the spider's method parse_item
|
||||
Rule(RegexLinkExtractor(allow=('item\.php', )), callback='parse_item'),
|
||||
)
|
||||
|
||||
def parse_item(self, response):
|
||||
log.msg('Hi, this is an item page! %s' % response.url)
|
||||
|
||||
hxs = HtmlXPathSelector(response)
|
||||
item = ScrapedItem()
|
||||
item.attribute('id', hxs.x('//td[@id="item_id"]/text()').re(r'ID: (\d+)'))
|
||||
item.attribute('name', hxs.x('//td[@id="item_name"]/text()'))
|
||||
item.attributE('description', hxs.x('//td[@id="item_description"]/text()'))
|
||||
return [item]
|
||||
|
||||
SPIDER = MySpider()
|
||||
|
||||
|
||||
This spider would start crawling example.com's home page, collecting category
|
||||
links, and item links, parsing the latter with the *parse_item* method. For
|
||||
each item response, some data will be extracted from the HTML using XPath, and
|
||||
a ScrapedItem will be filled with it.
|
||||
|
||||
|
||||
XMLFeedSpider
|
||||
=============
|
||||
|
||||
.. class:: XMLFeedSpider
|
||||
|
||||
XMLFeedSpider is designed for parsing XML feeds by iterating through them by a
|
||||
certain node name. The iterator can be chosen from: ``iternodes``, ``xml``,
|
||||
and ``html``. It's recommended to use the ``iternodes`` iterator for
|
||||
performance reasons, since the ``xml`` and ``html`` iterators generate the
|
||||
whole DOM at once in order to parse it. However, using ``html`` as the
|
||||
iterator may be useful when parsing XML with bad markup.
|
||||
|
||||
For setting the iterator and the tag name, you must define the following class
|
||||
attributes:
|
||||
|
||||
.. attribute:: XMLFeedSpider.iterator
|
||||
|
||||
A string which defines the iterator to use. It can be either:
|
||||
|
||||
- ``'iternodes'`` - a fast iterator based on regular expressions
|
||||
|
||||
- ``'html'`` - an iterator which uses HtmlXPathSelector. Keep in mind
|
||||
this uses DOM parsing and must load all DOM in memory which could be a
|
||||
problem for big feeds
|
||||
|
||||
- ``'xml'`` - an iterator which uses XmlXPathSelector. Keep in mind
|
||||
this uses DOM parsing and must load all DOM in memory which could be a
|
||||
problem for big feeds
|
||||
|
||||
It defaults to: ``'iternodes'``.
|
||||
|
||||
.. attribute:: XMLFeedSpider.itertag
|
||||
|
||||
A stirng with the name of the node (or element) to iterate in.
|
||||
|
||||
Apart from these new attributes, this spider has the following overrideable
|
||||
methods too:
|
||||
|
||||
.. method:: XMLFeedSpider.adapt_response(response)
|
||||
|
||||
A method that receives the response as soon as it arrives from the spider
|
||||
middleware and before start parsing it. It can be used used for modifying
|
||||
the response body before parsing it. This method receives a response and
|
||||
returns response (it could be the same or another one).
|
||||
|
||||
.. method:: XMLFeedSpider.parse_item(response, selector)
|
||||
|
||||
This method is called for the nodes matching the provided tag name
|
||||
(``itertag``). Receives the response and an XPathSelector for each node.
|
||||
Overriding this method is mandatory. Otherwise, you spider won't work.
|
||||
This method must return either a ScrapedItem, a Request, or a list
|
||||
containing any of them.
|
||||
|
||||
.. warning:: This method will soon change its name to ``parse_node``
|
||||
|
||||
.. method:: XMLFeedSpider.process_results(response, results)
|
||||
|
||||
This method is called for each result (item or request) returned by the
|
||||
spider, and it's intended to perform any last time processing required
|
||||
before returning the results to the framework core, for example setting the
|
||||
item IDs. It receives a list of results and the response which originated
|
||||
that results. It must return a list of results (Items or Requests)."""
|
||||
|
||||
|
||||
XMLFeedSpider example
|
||||
---------------------
|
||||
|
||||
These spiders are pretty easy to use, let's have at one example::
|
||||
|
||||
from scrapy import log
|
||||
from scrapy.contrib.spiders import XMLFeedSpider
|
||||
from scrapy.item import ScrapedItem
|
||||
|
||||
class MySpider(XMLFeedSpider):
|
||||
domain_name = 'example.com'
|
||||
start_urls = ['http://www.example.com/feed.xml']
|
||||
iterator = 'iternodes' # This is actually unnecesary, since it's the default value
|
||||
itertag = 'item'
|
||||
|
||||
def parse_item(self, response, node):
|
||||
log.msg('Hi, this is a <%s> node!: %s' % (self.itertag, ''.join(node.extract())))
|
||||
|
||||
item = ScrapedItem()
|
||||
item.attribute('id', node.x('@id'))
|
||||
item.attribute('name', node.x('name'))
|
||||
item.attribute('description', node.x('description'))
|
||||
return item
|
||||
|
||||
SPIDER = MySpider()
|
||||
|
||||
Basically what we did up there was creating a spider that downloads a feed from
|
||||
the given ``start_urls``, and then iterates through each of its ``item`` tags,
|
||||
prints them out, and stores some random data in ScrapedItems.
|
||||
|
||||
CSVFeedSpider
|
||||
=============
|
||||
|
||||
.. class:: XMLFeedSpider.CSVFeedSpider
|
||||
|
||||
.. warning:: The API of the XMLFeedSpider is not yet stable. Use with caution.
|
||||
|
||||
This spider is very similar to the XMLFeedSpider, although it iterates through
|
||||
rows, instead of nodes. It also has other two different attributes:
|
||||
|
||||
.. attribute:: XMLFeedSpider.delimiter
|
||||
|
||||
A string with the separator character for each field in the CSV file
|
||||
Defaults to ``','`` (comma).
|
||||
|
||||
.. attribute:: XMLFeedSpider.headers
|
||||
|
||||
A list of the rows contained in the file CSV feed which will be used for
|
||||
extracting fields from it.
|
||||
|
||||
In this spider, the method that gets called in each row iteration ``parse_row``
|
||||
instead of ``parse_item`` (like in :class:`XMLFeedSpider`).
|
||||
|
||||
.. method:: XMLFeedSpider.parse_row(response, row)
|
||||
|
||||
Receives a response and a dict (representing each row) with a key for each
|
||||
provided (or detected) header of the CSV file. This spider also gives the
|
||||
opportunity to override ``adapt_response`` and ``process_results`` methods
|
||||
for pre and post-processing purposes.
|
||||
|
||||
CSVFeedSpider example
|
||||
---------------------
|
||||
|
||||
Let's see an example similar to the previous one, but using CSVFeedSpider::
|
||||
|
||||
from scrapy import log
|
||||
from scrapy.contrib.spiders import CSVFeedSpider
|
||||
from scrapy.item import ScrapedItem
|
||||
|
||||
class MySpider(CSVFeedSpider):
|
||||
domain_name = 'example.com'
|
||||
start_urls = ['http://www.example.com/feed.csv']
|
||||
delimiter = ';'
|
||||
headers = ['id', 'name', 'description']
|
||||
|
||||
def parse_row(self, response, row):
|
||||
log.msg('Hi, this is a row!: %r' % row)
|
||||
|
||||
item = ScrapedItem()
|
||||
item.attribute('id', row['id'])
|
||||
item.attribute('name', row['name'])
|
||||
item.attribute('description', row['description'])
|
||||
return item
|
||||
|
||||
SPIDER = MySpider()
|
||||
|
||||
|
||||
|
|
@ -9,10 +9,11 @@ This section introduces all key concepts of Scrapy.
|
|||
:maxdepth: 1
|
||||
|
||||
architecture
|
||||
spiders
|
||||
selectors
|
||||
link-extractors
|
||||
items
|
||||
adaptors
|
||||
spiders
|
||||
item-pipeline
|
||||
downloader-middleware
|
||||
spider-middleware
|
||||
|
|
|
|||
|
|
@ -0,0 +1,25 @@
|
|||
.. _topics-link-extractors:
|
||||
|
||||
===============
|
||||
Link Extractors
|
||||
===============
|
||||
|
||||
.. module:: scrapy.link
|
||||
|
||||
LinkExtractors are objects whose purpose is to extract links from web pages.
|
||||
They're used in the :class:`~scrapy.contrib.spiders.CrawlSpider`, for defining
|
||||
crawling rules, among other places.
|
||||
|
||||
There are two different LinkExtractors available in Scrapy by default, but you
|
||||
create your own custom Link Extractor to suit your needs.
|
||||
|
||||
The only public method that every LinkExtractor has is ``extract_links``, which
|
||||
always receives a response, independently of which LinkExtractor are you using.
|
||||
This method should be called by you in case you want to extract links from a
|
||||
response yourself. In the case of rules, however, you'll only have to define
|
||||
your rules with the corresponding LinkExtractors, and the CrawlSpider will take
|
||||
care of extracting them for each response arriving.
|
||||
|
||||
See :ref:`ref-link-extractors` for the list of available built-in Link
|
||||
Extractors.
|
||||
|
||||
|
|
@ -4,267 +4,39 @@
|
|||
Spiders
|
||||
=======
|
||||
|
||||
.. module:: scrapy.spider
|
||||
Spiders are classes which define how a certain site (or domain) will be
|
||||
scraped, including how to crawl the site and how to extract scraped items from
|
||||
their pages. In other words, Spiders are the place where you define the custom
|
||||
behaviour for crawling and parsing pages for a particular site.
|
||||
|
||||
| Spiders are modules whose purpose is to scrape information from a certain domain.
|
||||
| It's there where you'll define the behaviour regarding the crawling and parsing processes, and where most of the action takes part, actually.
|
||||
For spiders, the scraping cycle goes through something like this:
|
||||
|
||||
In spiders, the scraping cycle goes through something like:
|
||||
1. You start by generating the initial Requests to crawl the first URLs, and
|
||||
specify a callback function to be called with the response downloaded from
|
||||
those requests.
|
||||
|
||||
1. Request for information.
|
||||
2. Find what you were looking for in the response you got.
|
||||
3. Adapt it (or not).
|
||||
4. Create items containing your scraped data.
|
||||
5. Store them (or print them, or whatever you want to do with them).
|
||||
The first requests to perform are obtained by calling the
|
||||
:meth:`BaseSpider.start_requests` method which (by default) generates
|
||||
:class:`~scrapy.http.Request` for the URLs specified in the
|
||||
:attr:`BaseSpider.start_urls` and the ``BaseSpider.parse`` method as
|
||||
callback function for the Requests.
|
||||
|
||||
| Now, this cycle starts by an entry point which you specify in the spider itself, and it's the first piece of information that you ask for.
|
||||
The ``start_urls`` and ``start_requests`` class attributes.
|
||||
| These attributes are expressed as lists containing either URLs as strings (for the ``start_urls``), or Request instances (for ``start_requests``).
|
||||
| It's not mandatory that you assign both attributes, but you should specify at least one of them.
|
||||
2. In the callback function you parse the response (web page) and return an
|
||||
iterable containing either ScrapedItem or Requests, or both. Those Requests
|
||||
will also contain a callback (maybe the same) and will then be followed by
|
||||
downloaded by Scrapy and then their response handled to the specified
|
||||
callback.
|
||||
|
||||
Another attribute that you must assign is ``domain_name``, which is nothing but a string containing the domain name of the site you're scraping.
|
||||
You have to you specify these attributes, because it's the way Scrapy knows which site is it crawling an where to start scraping from.
|
||||
3. In callback functions you parse the page contants, typically using
|
||||
:ref:`topics-selectors` (but you can also use BeautifuSoup, lxml or whatever
|
||||
mechanism you prefer) and generate items with the parsed data.
|
||||
|
||||
Now, although the previous applies for any kind of spider, there are different types of spiders, with different behaviours, let's check them out:
|
||||
4. Finally the items returned from the spider will be typically persisted in
|
||||
some Item pipeline.
|
||||
|
||||
BaseSpider
|
||||
----------
|
||||
|
||||
.. class:: BaseSpider
|
||||
|
||||
| This is the simplest available spider, and from which inherit any other spiders (either the ones that come built-in with Scrapy, or the ones users could make).
|
||||
| It doesn't provide any special functionality. It just requests the given ``start_urls``/``start_requests``, and calls the spider's method ``parse`` for each of the resulting responses.
|
||||
|
||||
.. attribute:: BaseSpider.domain_name
|
||||
|
||||
Identifies the spider. It must be unique, that is, you can't set the same
|
||||
domain name for different spiders.
|
||||
|
||||
.. attribute:: BaseSpider.start_urls
|
||||
|
||||
Is a list of URLs where the spider will begin to crawl from. So, the first
|
||||
pages downloaded will be those listed here. The subsequent URLs will be
|
||||
generated successively from data contained in the start URLs.
|
||||
|
||||
.. attribute:: BaseSpider.start_requests
|
||||
|
||||
.. method:: BaseSpider.parse (response)
|
||||
|
||||
Iis the callback method of the spider. This means that each time a URL is
|
||||
retrieved, the downloaded data (response) will be passed to this method.
|
||||
|
||||
The ``parse`` method is in charge of processing the response and returning
|
||||
scraped data and or more URLs to follow, because of this, the method must
|
||||
always return a list or at least an empty one.
|
||||
|
||||
Let's see an example::
|
||||
|
||||
from scrapy import log # This module is useful for printing out debug information
|
||||
from scrapy.spider import BaseSpider
|
||||
|
||||
class MySpider(BaseSpider):
|
||||
domain_name = 'http://www.example.com'
|
||||
start_urls = [
|
||||
'http://www.example.com/1.html',
|
||||
'http://www.example.com/2.html',
|
||||
'http://www.example.com/3.html',
|
||||
]
|
||||
|
||||
def parse(self, response):
|
||||
log.msg('Hey! A response from %s has just arrived!' % response.url)
|
||||
return []
|
||||
|
||||
SPIDER = MySpider()
|
||||
|
||||
.. module:: scrapy.contrib.spiders
|
||||
|
||||
CrawlSpider
|
||||
-----------
|
||||
|
||||
.. class:: CrawlSpider
|
||||
|
||||
| This is the most commonly used spider, and it's the one who crawls over HTML pages, extracts links from there (given certain rules of extraction), and scrapes items from
|
||||
there.
|
||||
| This spider is a bit more complicated than the previous one, because it introduces a few new concepts, but you'll probably find it useful.
|
||||
|
|
||||
| Apart from the attributes inherited from BaseSpider (that you **must** specify), this class provides you with a new attribute: ``rules``.
|
||||
| This one is a tuple containing one or more ``Rule`` objects.
|
||||
| Each ``Rule`` defines a certain behaviour for crawling the site, by the following parameters (the ones between [ ] are optional):
|
||||
|
||||
* link_extractor: A ``LinkExtractor`` instance, the one that will take care of extracting urls from each response (i'll explain this further).
|
||||
* [callback]: A callable, or a string (in which case a method from the spider class with that name will be used) to be called for each link extracted
|
||||
with the link_extractor. This callback must always return a list, which can contain both ScrapedItems (or any descendant), and Requests.
|
||||
* [cb_kwargs]: A dictionary containing the keyword arguments to be passed to the callback function.
|
||||
* [follow]: A boolean defining whether links should be followed from each response extracted with this rule or not. If callback was specified, defaults
|
||||
to False, else defaults to True.
|
||||
* [process_links]: A callable, or a string (applying the same as in the callback) to be called with each set of links extracted from each response with
|
||||
the link_extractor (mainly for filtering purposes).
|
||||
|
||||
LinkExtractors
|
||||
^^^^^^^^^^^^^^
|
||||
|
||||
| LinkExtractors are objects designed -obviously- for extracting links from web pages.
|
||||
| There are currently only two different LinkExtractors available in Scrapy: ``LinkExtractor`` and ``RegexLinkExtractor``.
|
||||
| The first one extracts links from a response with the given tag names and attributes. It doesn't do any other filtering.
|
||||
| RegexLinkExtractors, however, extract links from a response by applying several filters that you can specify, mostly regular expressions that match (or not)
|
||||
the extracted links.
|
||||
| These are the parameters that LinkExtractors may receive when instanciating them:
|
||||
|
||||
.. class:: scrapy.link.LinkExtractor
|
||||
|
||||
* tag: Can be either a tag name in a string, or a function that receives a tag name and returns True if links should be extracted from it, or False if they
|
||||
shouldn't. Defaults to 'a'.
|
||||
* attr: The same as in ``tag``, for attribute names.
|
||||
* unique: A boolean that decides whether links with the same url should be extracted only once or not.
|
||||
|
||||
.. class:: scrapy.link.extractors.RegexLinkExtractor
|
||||
|
||||
* tag: The same purpose as in LinkExtractor.
|
||||
* attr: The same purpose as in LinkExtractor.
|
||||
* unique: The same purpose as in LinkExtractor.
|
||||
* allow: A list of regular expressions that the (absolute) urls must match in order to be extracted.
|
||||
* deny: A list of regular expressions that makes any url matching them be ignored.
|
||||
* allow_domains: A list of domains from which to extract urls.
|
||||
* deny_domains: A list of domains to not extract urls from.
|
||||
* restrict_xpaths: Only extract links from the areas inside the provided xpaths (in a list).
|
||||
* tags: List of tags to extract links from. Defaults to ('a', 'area').
|
||||
* attrs: List of attributes to extract links from. Defaults to ('href', )
|
||||
* canonicalize: Canonicalize each extracted url (using scrapy.utils.url.canonicalize_url). Defaults to True.
|
||||
|
||||
| The only public method that every LinkExtractor has is ``extract_links``, which always receives a response, independently of which LinkExtractor are you using.
|
||||
This method should be called by you in case you want to extract links from a response yourself.
|
||||
| In the case of rules, however, you'll only have to define your rules with the corresponding LinkExtractors,
|
||||
and the CrawlSpider will take care of extracting them for each response arriving.
|
||||
|
||||
Let's now take a look at an example CrawlSpider with rules::
|
||||
|
||||
from scrapy import log
|
||||
from scrapy.contrib.spiders import CrawlSpider, Rule
|
||||
from scrapy.link.extractors import RegexLinkExtractor
|
||||
from scrapy.xpath.selector import HtmlXPathSelector
|
||||
from scrapy.item import ScrapedItem
|
||||
|
||||
class MySpider(CrawlSpider):
|
||||
domain_name = 'example.com'
|
||||
start_urls = ['http://www.example.com']
|
||||
|
||||
rules = (
|
||||
# Extract links matching 'category.php' (but not matching 'subsection.php') and follow links from them (since no callback means follow=True by default).
|
||||
Rule(RegexLinkExtractor(allow=('category\.php', ), deny=('subsection\,php', ))),
|
||||
|
||||
# Extract links matching 'item.php' and parse them with the spider's method parse_item
|
||||
Rule(RegexLinkExtractor(allow=('item\.php', )), callback='parse_item'),
|
||||
)
|
||||
|
||||
def parse_item(self, response):
|
||||
log.msg('Hi, this is an item page! %s' % response.url)
|
||||
|
||||
hxs = HtmlXPathSelector(response)
|
||||
item = ScrapedItem()
|
||||
item.attribute('id', hxs.x('//td[@id="item_id"]/text()').re(r'ID: (\d+)'))
|
||||
item.attribute('name', hxs.x('//td[@id="item_name"]/text()'))
|
||||
item.attributE('description', hxs.x('//td[@id="item_description"]/text()'))
|
||||
return [item]
|
||||
|
||||
SPIDER = MySpider()
|
||||
|
||||
|
||||
This spider would start crawling example.com's home page, collecting category links, and item links, parsing the latter with the *parse_item* method.
|
||||
For each item response, some data will be extracted from the HTML using XPath, and a ScrapedItem will be filled with it.
|
||||
|
||||
Feed Spiders
|
||||
-------------
|
||||
|
||||
XMLFeedSpider
|
||||
^^^^^^^^^^^^^
|
||||
|
||||
.. class:: XMLFeedSpider
|
||||
|
||||
XMLFeedSpider is designed for parsing XML feeds by iterating through them by a certain node name.
|
||||
The iterator can be chosen from: ``iternodes``, ``xml``, and ``html``.
|
||||
It's recommended to use the ``iternodes`` iterator for performance reasons, since the ``xml``
|
||||
and ``html`` iterators generate the whole DOM at once in order to parse it.
|
||||
However, using ``html`` as the iterator may be useful when parsing XML with bad markup.
|
||||
|
||||
For setting the iterator and the tag name, you must define the class attributes
|
||||
``iterator`` and ``itertag``.
|
||||
The default values are ``iternodes`` for ``iterator``, and ``item`` for ``itertag``.
|
||||
|
||||
Apart from these new attributes, this spider has some new overrideable methods too:
|
||||
|
||||
* adapt_response: used for modifying the response and/or its body before parsing it.
|
||||
Receives a response and returns another one.
|
||||
* parse_item: the method to be called for the nodes matching the provided tag name (``itertag``).
|
||||
Receives the response and an XPathSelector for each node.
|
||||
Overriding this method is mandatory. If not, the spider won't work.
|
||||
This method must return either a ScrapedItem, a Request, or a list containing any of them.
|
||||
* process_results: this method will be called after each call of parse_node, with a response
|
||||
and the parsing list of results.
|
||||
|
||||
These spiders are pretty easy to use, let's have a look::
|
||||
|
||||
from scrapy import log
|
||||
from scrapy.contrib.spiders import XMLFeedSpider
|
||||
from scrapy.item import ScrapedItem
|
||||
|
||||
class MySpider(XMLFeedSpider):
|
||||
domain_name = 'example.com'
|
||||
start_urls = ['http://www.example.com/feed.xml']
|
||||
iterator = 'iternodes' # This is actually unnecesary, since it's the default value
|
||||
itertag = 'item'
|
||||
|
||||
def parse_item(self, response, node):
|
||||
log.msg('Hi, this is a <%s> node!: %s' % (self.itertag, ''.join(node.extract())))
|
||||
|
||||
item = ScrapedItem()
|
||||
item.attribute('id', node.x('@id'))
|
||||
item.attribute('name', node.x('name'))
|
||||
item.attribute('description', node.x('description'))
|
||||
return item
|
||||
|
||||
SPIDER = MySpider()
|
||||
|
||||
Basically what we did up there was creating a spider that downloads a feed from the given ``start_urls``,
|
||||
iterates through each of its 'item' tags, prints them out, and stores some random data in ScrapedItems.
|
||||
|
||||
CSVFeedSpider
|
||||
^^^^^^^^^^^^^
|
||||
|
||||
.. class:: CSVFeedSpider
|
||||
|
||||
This spider is very similar to the XMLFeedSpider, although it iterates through rows, instead of nodes.
|
||||
It also has other two different attributes: ``delimiter``, and ``headers``.
|
||||
The ``delimiter`` is a string representing the limit between each field in the CSV file,
|
||||
while the ``headers`` are an ordered list of field names (in strings) that the file contains.
|
||||
|
||||
The default ``delimiter`` is the same as in Python's csv module, a `,` (comma), while the ``headers`` parameter,
|
||||
if not specified, is tried to be found out.
|
||||
|
||||
In this case, the method that gets called in each row iteration is called ``parse_row`` instead of ``parse_item`` (as it was in XMLFeedSpider),
|
||||
and receives a response and a dictionary (representing each row) with a key for each provided (or detected) header of the CSV file.
|
||||
This spider also gives the opportunity to override ``adapt_response`` and ``process_results`` methods for pre/post-processing purposes.
|
||||
|
||||
Let's see an example similar to the previous one, but using CSVFeedSpider::
|
||||
|
||||
from scrapy import log
|
||||
from scrapy.contrib.spiders import CSVFeedSpider
|
||||
from scrapy.item import ScrapedItem
|
||||
|
||||
class MySpider(CSVFeedSpider):
|
||||
domain_name = 'example.com'
|
||||
start_urls = ['http://www.example.com/feed.csv']
|
||||
delimiter = ';'
|
||||
headers = ['id', 'name', 'description']
|
||||
|
||||
def parse_row(self, response, row):
|
||||
log.msg('Hi, this is a row!: %r' % row)
|
||||
|
||||
item = ScrapedItem()
|
||||
item.attribute('id', row['id'])
|
||||
item.attribute('name', row['name'])
|
||||
item.attribute('description', row['description'])
|
||||
return item
|
||||
|
||||
SPIDER = MySpider()
|
||||
Even though this cycles applies (more or less) to any kind of spider, there are
|
||||
different kind of default spiders bundled into Scrapy for different purposes.
|
||||
We will talk about those types here.
|
||||
|
||||
See :ref:`ref-spiders` for the list of default spiders available in Scrapy.
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue