diff --git a/docs/intro/overview.rst b/docs/intro/overview.rst index c30963db8..9a3015ddc 100644 --- a/docs/intro/overview.rst +++ b/docs/intro/overview.rst @@ -13,172 +13,83 @@ precisely, `web scraping`_), it can also be used to extract data using APIs (such as `Amazon Associates Web Services`_) or as a general purpose web crawler. -The purpose of this document is to introduce you to the concepts behind Scrapy -so you can get an idea of how it works and decide if Scrapy is what you need. -When you're ready to start a project, you can :ref:`start with the tutorial -`. +Walk-through of an example spider +================================= -Pick a website -============== +In order to show you what Scrapy brings to the table, we'll walk you +through an example of a Scrapy Spider using the simplest way to run a spider. -So you need to extract some information from a website, but the website doesn't -provide any API or mechanism to access that info programmatically. Scrapy can -help you extract that information. +Once you're ready to dive in more, you can :ref:`follow the tutorial +and build a full-blown Scrapy project `. -Let's say we want to extract the URL, name, description and size of all torrent -files added today in the `Mininova`_ site. - -The list of all torrents added today can be found on this page: - - http://www.mininova.org/today - -.. _intro-overview-item: - -Define the data you want to scrape -================================== - -The first thing is to define the data we want to scrape. In Scrapy, this is -done through :ref:`Scrapy Items ` (Torrent files, in this case). - -This would be our Item:: +So, here's the code for a spider that follows the links to the top +voted questions on StackOverflow and scrapes some data from each page:: import scrapy - class TorrentItem(scrapy.Item): - url = scrapy.Field() - name = scrapy.Field() - description = scrapy.Field() - size = scrapy.Field() -Write a Spider to extract the data -================================== + class StackOverflowSpider(scrapy.Spider): + name = 'stackoverflow' + start_urls = ['http://stackoverflow.com/questions?sort=votes'] -The next thing is to write a Spider which defines the start URL -(http://www.mininova.org/today), the rules for following links and the rules -for extracting the data from pages. + def parse(self, response): + for href in response.css('.question-summary h3 a::attr(href)'): + full_url = response.urljoin(href.extract()) + yield scrapy.Request(full_url, callback=self.parse_question) -If we take a look at that page content we'll see that all torrent URLs are like -``http://www.mininova.org/tor/NUMBER`` where ``NUMBER`` is an integer. We'll use -that to construct the regular expression for the links to follow: ``/tor/\d+``. - -We'll use `XPath`_ for selecting the data to extract from the web page HTML -source. Let's take one of those torrent pages: - - http://www.mininova.org/tor/2676093 - -And look at the page HTML source to construct the XPath to select the data we -want which is: torrent name, description and size. - -.. highlight:: html - -By looking at the page HTML source we can see that the file name is contained -inside a ``

`` tag:: - -

Darwin - The Evolution Of An Exhibition

- -.. highlight:: none - -An XPath expression to extract the name could be:: - - //h1/text() - -.. highlight:: html - -And the description is contained inside a ``
`` tag with ``id="description"``:: - -

Description:

- -
- Short documentary made for Plymouth City Museum and Art Gallery regarding the setup of an exhibit about Charles Darwin in conjunction with the 200th anniversary of his birth. - - ... - -.. highlight:: none - -An XPath expression to select the description could be:: - - //div[@id='description'] - -.. highlight:: html - -Finally, the file size is contained in the second ``

`` tag inside the ``

`` -tag with ``id=specifications``:: - -
- -

- Category: - Movies > Documentary -

- -

- Total size: - 150.62 megabyte

+ def parse_question(self, response): + title = response.css('h1 a::text').extract_first() + votes = response.css('.question .vote-count-post::text').extract_first() + tags = response.css('.question .post-tag::text').extract() + body = response.css('.question .post-text').extract_first() + yield { + 'title': title, + 'votes': votes, + 'body': body, + 'tags': tags, + 'link': response.url, + } -.. highlight:: none +Put this in a file, name it to something like ``stackoverflow_spider.py`` +and run the spider using the :command:`runspider` command:: -An XPath expression to select the file size could be:: + scrapy runspider stackoverflow_spider.py -o top-stackoverflow-questions.json - //div[@id='specifications']/p[2]/text()[2] -.. highlight:: python +When this finishes you will have in the ``top-stackoverflow-questions.json`` file +a list of the most upvoted questions in StackOverflow in JSON format, containing the +title, link, number of upvotes, a list of the tags and the question content in HTML. -For more information about XPath see the `XPath reference`_. -Finally, here's the spider code:: +What just happened? +------------------- - from scrapy.contrib.spiders import CrawlSpider, Rule - from scrapy.contrib.linkextractors import LinkExtractor +When you ran the command ``scrapy runspider somefile.py``, Scrapy looked +for a Spider definition inside it and ran it through its crawler engine. - class MininovaSpider(CrawlSpider): +The crawl started by making requests to the URLs defined in the ``start_urls`` +attribute (in this case, only the URL for StackOverflow top questions page), +and then called the default callback method ``parse`` passing the response +object as an argument. - name = 'mininova' - allowed_domains = ['mininova.org'] - start_urls = ['http://www.mininova.org/today'] - rules = [Rule(LinkExtractor(allow=['/tor/\d+']), 'parse_torrent')] +In the ``parse`` callback, we scrape the links to the questions and +yield a few more requests to be processed, registering for them +the method ``parse_question`` as the callback to be called when the +requests are complete. - def parse_torrent(self, response): - torrent = TorrentItem() - torrent['url'] = response.url - torrent['name'] = response.xpath("//h1/text()").extract() - torrent['description'] = response.xpath("//div[@id='description']").extract() - torrent['size'] = response.xpath("//div[@id='info-left']/p[2]/text()[2]").extract() - return torrent +Finally, the ``parse_question`` callback scrapes the question data +for each page yielding a dict, which Scrapy then collects and +writes to a JSON file as requested in the command line. -The ``TorrentItem`` class is :ref:`defined above `. +.. note:: -Run the spider to extract the data -================================== + This is using :ref:`feed exports ` to generate the + JSON file, you can easily change the export format (XML or CSV, for example) or the + storage backend (FTP or `Amazon S3`_, for example). You can also write an + :ref:`item pipeline ` to store the items in a database. -Finally, we'll run the spider to crawl the site and output the file -``scraped_data.json`` with the scraped data in JSON format:: - - scrapy crawl mininova -o scraped_data.json - -This uses :ref:`feed exports ` to generate the JSON file. -You can easily change the export format (XML or CSV, for example) or the -storage backend (FTP or `Amazon S3`_, for example). - -You can also write an :ref:`item pipeline ` to store the -items in a database very easily. - -Review scraped data -=================== - -If you check the ``scraped_data.json`` file after the process finishes, you'll -see the scraped items there:: - - [{"url": "http://www.mininova.org/tor/2676093", "name": ["Darwin - The Evolution Of An Exhibition"], "description": ["Short documentary made for Plymouth ..."], "size": ["150.62 megabyte"]}, - # ... other items ... - ] - -You'll notice that all field values (except for the ``url`` which was assigned -directly) are actually lists. This is because the :ref:`selectors -` return lists. You may want to store single values, or -perform some additional parsing/cleansing to the values. That's what -:ref:`Item Loaders ` are for. .. _topics-whatelse: @@ -189,68 +100,37 @@ You've seen how to extract and store items from a website using Scrapy, but this is just the surface. Scrapy provides a lot of powerful features for making scraping easy and efficient, such as: -* Built-in support for :ref:`selecting and extracting ` data - from HTML and XML sources - -* Built-in support for cleaning and sanitizing the scraped data using a - collection of reusable filters (called :ref:`Item Loaders `) - shared between all the spiders. +* An :ref:`interactive shell console ` (IPython aware) for trying + out the CSS and XPath expressions to scrape data, very useful when writing or + debugging your spiders. * Built-in support for :ref:`generating feed exports ` in multiple formats (JSON, CSV, XML) and storing them in multiple backends (FTP, S3, local filesystem) -* A media pipeline for :ref:`automatically downloading images ` - (or any other media) associated with the scraped items - -* Support for :ref:`extending Scrapy ` by plugging - your own functionality using :ref:`signals ` and a - well-defined API (middlewares, :ref:`extensions `, and - :ref:`pipelines `). - -* Wide range of built-in middlewares and extensions for: - - * cookies and session handling - * HTTP compression - * HTTP authentication - * HTTP cache - * user-agent spoofing - * robots.txt - * crawl depth restriction - * and more - * Robust encoding support and auto-detection, for dealing with foreign, non-standard and broken encoding declarations. -* Support for creating spiders based on pre-defined templates, to speed up - spider creation and make their code more consistent on large projects. See - :command:`genspider` command for more details. - -* Extensible :ref:`stats collection ` for multiple spider - metrics, useful for monitoring the performance of your spiders and detecting - when they get broken - -* An :ref:`Interactive shell console ` for trying XPaths, very - useful for writing and debugging your spiders - -* A :ref:`System service ` designed to ease the deployment and - run of your spiders in production. +* Strong :ref:`extensibility support ` and lots of built-in + extensions and middlewares to handle things like cookies, crawl throttling, + HTTP caching, HTTP compression, user-agent spoofing, robots.txt, + stats collection and many more. * A :ref:`Telnet console ` for hooking into a Python console running inside your Scrapy process, to introspect and debug your crawler -* :ref:`Logging ` facility that you can hook on to for catching - errors during the scraping process. +* A caching DNS resolver * Support for crawling based on URLs discovered through `Sitemaps`_ -* A caching DNS resolver +* A media pipeline for :ref:`automatically downloading images ` + (or any other media) associated with the scraped items What's next? ============ -The next obvious steps are for you to `download Scrapy`_, read :ref:`the +The next obvious steps for you are to `download Scrapy`_, read :ref:`the tutorial ` and join `the community`_. Thanks for your interest! @@ -258,9 +138,6 @@ interest! .. _the community: http://scrapy.org/community/ .. _screen scraping: http://en.wikipedia.org/wiki/Screen_scraping .. _web scraping: http://en.wikipedia.org/wiki/Web_scraping -.. _Amazon Associates Web Services: https://affiliate-program.amazon.com/gp/advertising/api/detail/main.html -.. _Mininova: http://www.mininova.org -.. _XPath: http://www.w3.org/TR/xpath -.. _XPath reference: http://www.w3.org/TR/xpath +.. _Amazon Associates Web Services: http://aws.amazon.com/associates/ .. _Amazon S3: http://aws.amazon.com/s3/ .. _Sitemaps: http://www.sitemaps.org