diff --git a/scrapy/trunk/docs/proposed/index.rst b/scrapy/trunk/docs/proposed/index.rst index 07fb4678f..f9063bcee 100644 --- a/scrapy/trunk/docs/proposed/index.rst +++ b/scrapy/trunk/docs/proposed/index.rst @@ -16,3 +16,7 @@ Proposed documentation :maxdepth: 1 tutorial + +.. toctree:: + + spiders diff --git a/scrapy/trunk/docs/proposed/introduction.rst b/scrapy/trunk/docs/proposed/introduction.rst index 25204d6db..602f8d3ad 100644 --- a/scrapy/trunk/docs/proposed/introduction.rst +++ b/scrapy/trunk/docs/proposed/introduction.rst @@ -4,8 +4,8 @@ Introduction .. architecture: -Architecture -============ +Overview +======== .. image:: _images/scrapy_architecture.png :width: 700 @@ -15,7 +15,7 @@ Architecture .. _items: Items -===== +----- In Scrapy, Items are the placeholder to use for the scraped data. They are represented by a :class:`~scrapy.item.ScrapedItem` object, or any subclass @@ -24,7 +24,7 @@ instance, and store the information in instance attributes. .. _request-response: Requests and Responses -====================== +---------------------- Scrapy uses :class:`~scrapy.http.Request` and :class:`~scrapy.http.Response` objects for crawling web sites. @@ -36,10 +36,10 @@ Downloader, which actually executes the request and returns a :class:`~scrapy.http.Response` object to the :class:`Request's callback function `. -.. _spiders: +.. _overview-spiders: Spiders -======= +------- Spiders are user written classes which define how a certain site (or domain) will be scraped; including how to crawl the site and how to scrape :ref:`Items @@ -48,47 +48,10 @@ will be scraped; including how to crawl the site and how to scrape :ref:`Items All Spiders must be descendant of :class:`~scrapy.spider.BaseSpider` or any subclass of it, in :ref:`ref-spiders` you can see a list of available Spiders in Scrapy. - -Scraping cycle --------------- - -1. *Generating Requests*: - - The first step is to generate the initial :ref:`Requests - ` to crawl the first URLs, and specify a callback - function to be called with the :ref:`Response ` - downloaded from those :ref:`Requests `. - - The first :ref:`Requests `. to perform are obtained by - calling the :meth:`~scrapy.spider.BaseSpider.start_requests` method which - (by default) generates :class:`~scrapy.http.Request` for the URLs specified - in the :attr:`BaseSpider.start_urls` and the - :meth:`~scrapy.spider.BaseSpider.parse` method as callback function for the - :ref:`Requests `.. - -2. *Parsing Responses*: - - In callback functions you parse the :ref:`Response ` - contents and return an iterable object containing :ref:`Items `, - :ref:`Requests `, or both. - - Typically you do the parsing by using :ref:`selectors`, but you could also - use BeautifuSoup, lxml or the mechanism of your choice. - -3. *The final step*: - - Returned :ref:`Requests ` (if any) will be downloaded by - Scrapy and their :ref:`Responses ` handled to the - specified callback, wich could be the same than the one specified in the - first step. - - Returned :ref:`Items `. (if any) will be directed to the :ref:`Item - Pipeline `. - .. _selectors: Selectors -========= +--------- Selectors are the recommended tool to extract information from documents. They retrieve information from the :ref:`Response ` body using @@ -108,22 +71,16 @@ node, or the entire document. .. _item-pipeline: Item Pipeline -============= +------------- After an :ref:`Item ` has been scraped by a :ref:`Spider `, it is sent to the Item Pipeline which allows us to perform some actions over the :ref:`scrapped Items `. -The Item Pipeline is a list of user written Python classes that define the -:meth:`process_item` method, which is called sequentially for every element. +The Item Pipeline is a list of user written Python classes that implement a +specific method , which is called sequentially for every element of the +Pipeline. -The :meth:`process_item` must return the Item object on a successful action, -or raise a :exception:`DropItem` exception (ex: failing a validation test). -Dropped :ref:`Items ` are no longer processed by further pipeline -components. - -Typical uses of the Item Pipeline include: - -* Clean the HTML in Item attributes -* Validate the Item -* Store the Item +Each element receives the Scraped Item, do an action upon it (like validating, +checking for duplicates, store the item), and then decide if the Item +continues trough the Pipeline or the item is dropped. diff --git a/scrapy/trunk/docs/proposed/spiders.rst b/scrapy/trunk/docs/proposed/spiders.rst new file mode 100644 index 000000000..9cc6154ce --- /dev/null +++ b/scrapy/trunk/docs/proposed/spiders.rst @@ -0,0 +1,229 @@ +.. _spiders: + +Spiders +======= + +Spiders are user written classes which define how a certain site (or domain) +will be scraped; including how to crawl the site and how to scrape :ref:`Items +` from their pages. + +All Spiders must be descendant of :class:`~scrapy.spider.BaseSpider` or any +subclass of it, below you can see a list of available Spiders in Scrapy. + +.. _spiders-ref: + +Available Spiders +================= + +.. module:: scrapy.spider + +BaseSpider +---------- + +.. autoclass:: BaseSpider(object) + :members: + + .. attribute:: domain_name + + A string which defines the domain name for this spider, which will also + be the unique identifier for this spider (which means you can't have two + spider with the same :attr:`domain_name`). This is the most important + spider attribute and it's required, and it's the name by which Scrapy + will known the spider. + + .. attribute:: extra_domain_names + + An optional list of strings containing additional domains that this + spider is allowed to crawl. Requests for URLs not belonging to the domain + name specified in :attr:`domain_name` or this list won't be followed. + + .. attribute:: start_urls + + Is a list of URLs where the spider will begin to crawl from, when no + particular URLs are specified. So, the first pages downloaded will be + those listed here. The subsequent URLs will be generated successively + from data contained in the start URLs. + +BaseSpider example +^^^^^^^^^^^^^^^^^^ + +Let's see an example:: + + from scrapy import log # This module is useful for printing out debug information + from scrapy.spider import BaseSpider + + class MySpider(BaseSpider): + domain_name = 'http://www.example.com' + start_urls = [ + 'http://www.example.com/1.html', + 'http://www.example.com/2.html', + 'http://www.example.com/3.html', + ] + + def parse(self, response): + log.msg('Hey! A response from %s has just arrived!' % response.url) + return [] + + SPIDER = MySpider() + +.. module:: scrapy.contrib.spiders + +CrawlSpider +----------- + +.. autoclass:: CrawlSpider(BaseSpider) + :members: + + .. attribute:: CrawlSpider.rules + + Which is a list of one (or more) :class:`Rule` objects. Each + :class:`Rule` defines a certain behaviour for crawling the site. Rules + objects are described below. + +Crawling rules +^^^^^^^^^^^^^^ + +.. autoclass:: Rule(link_extractor, callback=None, cb_kwargs=None, follow=None, process_links=None) + +CrawlSpider example +^^^^^^^^^^^^^^^^^^^ + +Let's now take a look at an example CrawlSpider with Rules:: + + from scrapy import log + from scrapy.contrib.spiders import CrawlSpider, Rule + from scrapy.link.extractors import RegexLinkExtractor + from scrapy.xpath.selector import HtmlXPathSelector + from scrapy.item import ScrapedItem + + class MySpider(CrawlSpider): + domain_name = 'example.com' + start_urls = ['http://www.example.com'] + + rules = ( + # Extract links matching 'category.php' (but not matching 'subsection.php') + # and follow links from them (since no callback means follow=True by default). + Rule(RegexLinkExtractor(allow=('category\.php', ), deny=('subsection\,php', ))), + + # Extract links matching 'item.php' and parse them with the spider's method parse_item + Rule(RegexLinkExtractor(allow=('item\.php', )), callback='parse_item'), + ) + + def parse_item(self, response): + log.msg('Hi, this is an item page! %s' % response.url) + + hxs = HtmlXPathSelector(response) + item = ScrapedItem() + item.attribute('id', hxs.x('//td[@id="item_id"]/text()').re(r'ID: (\d+)')) + item.attribute('name', hxs.x('//td[@id="item_name"]/text()')) + item.attributE('description', hxs.x('//td[@id="item_description"]/text()')) + return [item] + + SPIDER = MySpider() + +This spider would start crawling example.com's home page, collecting category +links, and item links, parsing the latter with the *parse_item* method. For +each item response, some data will be extracted from the HTML using XPath, and +a ScrapedItem will be filled with it. + +XMLFeedSpider +------------- + +.. autoclass:: XMLFeedSpider(BaseSpider) + :members: + + .. attribute:: iterator + + A string which defines the iterator to use. It can be either: + + * 'iternodes': a fast iterator based on regular expressions + + * 'html': an iterator which uses HtmlXPathSelector. Keep in mind this + uses DOM parsing and must load all DOM in memory which could be a + problem for big feeds + + * 'xml': an iterator which uses XmlXPathSelector. Keep in mind this uses + DOM parsing and must load all DOM in memory which could be a problem + for big feeds + + It defaults to: 'iternodes'. + + .. attribute:: itertag + + A string with the name of the node (or element) to iterate in. + + +XMLFeedSpider example +^^^^^^^^^^^^^^^^^^^^^ + +These spiders are pretty easy to use, let's have at one example:: + + from scrapy import log + from scrapy.contrib.spiders import XMLFeedSpider + from scrapy.item import ScrapedItem + + class MySpider(XMLFeedSpider): + domain_name = 'example.com' + start_urls = ['http://www.example.com/feed.xml'] + iterator = 'iternodes' # This is actually unnecesary, since it's the default value + itertag = 'item' + + def parse_nodes(self, response, node): + log.msg('Hi, this is a <%s> node!: %s' % (self.itertag, ''.join(node.extract()))) + + item = ScrapedItem() + item.attribute('id', node.x('@id')) + item.attribute('name', node.x('name')) + item.attribute('description', node.x('description')) + return item + + SPIDER = MySpider() + +Basically what we did up there was creating a spider that downloads a feed from +the given ``start_urls``, and then iterates through each of its ``item`` tags, +prints them out, and stores some random data in ScrapedItems. + +CSVFeedSpider +------------- + +.. warning:: The API of the CSVFeedSpider is not yet stable. Use with caution. + +.. autoclass:: CSVFeedSpider(BaseSpider) + :members: + + .. attribute:: CSVFeedSpider.delimiter + + A string with the separator character for each field in the CSV file + Defaults to ``','`` (comma). + + .. attribute:: CSVFeedSpider.headers + + A list of the rows contained in the file CSV feed which will be used for + extracting fields from it. + +CSVFeedSpider example +^^^^^^^^^^^^^^^^^^^^^ + +Let's see an example similar to the previous one, but using CSVFeedSpider:: + + from scrapy import log + from scrapy.contrib.spiders import CSVFeedSpider + from scrapy.item import ScrapedItem + + class MySpider(CSVFeedSpider): + domain_name = 'example.com' + start_urls = ['http://www.example.com/feed.csv'] + delimiter = ';' + headers = ['id', 'name', 'description'] + + def parse_rows(self, response, row): + log.msg('Hi, this is a row!: %r' % row) + + item = ScrapedItem() + item.attribute('id', row['id']) + item.attribute('name', row['name']) + item.attribute('description', row['description']) + return item + + SPIDER = MySpider() +