RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/docs/intro/overview.rst

6.5 KiB
Raw Blame History

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> </head>

Scrapy at a glance

Scrapy (/ˈskreɪpaɪ/) is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful applications, like data mining, information processing or historical archival.

Even though Scrapy was originally designed for web scraping, it can also be used to extract data using APIs (such as Amazon Associates Web Services) or as a general purpose web crawler.

Walk-through of an example spider

In order to show you what Scrapy brings to the table, we'll walk you through an example of a Scrapy Spider using the simplest way to run a spider.

Here's the code for a spider that scrapes famous quotes from website https://quotes.toscrape.com, following the pagination:

System Message: WARNING/2 (<stdin>, line 25)

Cannot analyze code. Pygments package not found.

.. code-block:: python

    import scrapy


    class QuotesSpider(scrapy.Spider):
        name = "quotes"
        start_urls = [
            "https://quotes.toscrape.com/tag/humor/",
        ]

        def parse(self, response):
            for quote in response.css("div.quote"):
                yield {
                    "author": quote.xpath("span/small/text()").get(),
                    "text": quote.css("span.text::text").get(),
                }

            next_page = response.css('li.next a::attr("href")').get()
            if next_page is not None:
                yield response.follow(next_page, self.parse)

Put this in a text file, name it something like quotes_spider.py and run the spider using the :command:`runspider` command:

System Message: ERROR/3 (<stdin>, line 47); backlink

Unknown interpreted text role "command".
scrapy runspider quotes_spider.py -o quotes.jsonl

When this finishes you will have in the quotes.jsonl file a list of the quotes in JSON Lines format, containing the text and author, which will look like this:

{"author": "Jane Austen", "text": "\u201cThe person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.\u201d"}
{"author": "Steve Martin", "text": "\u201cA day without sunshine is like, you know, night.\u201d"}
{"author": "Garrison Keillor", "text": "\u201cAnyone who thinks sitting in church can make you a Christian must also think that sitting in a garage can make you a car.\u201d"}
...

What just happened?

When you ran the command scrapy runspider quotes_spider.py, Scrapy looked for a Spider definition inside it and ran it through its crawler engine.

The crawl started by making requests to the URLs defined in the start_urls attribute (in this case, only the URL for quotes in the humor category) and called the default callback method parse, passing the response object as an argument. In the parse callback, we loop through the quote elements using a CSS Selector, yield a Python dict with the extracted quote text and author, look for a link to the next page and schedule another request using the same parse method as callback.

Here you will notice one of the main advantages of Scrapy: requests are :ref:`scheduled and processed asynchronously <topics-architecture>`. This means that Scrapy doesn't need to wait for a request to be finished and processed, it can send another request or do other things in the meantime. This also means that other requests can keep going even if a request fails or an error happens while handling it.

System Message: ERROR/3 (<stdin>, line 75); backlink

Unknown interpreted text role "ref".

While this enables you to do very fast crawls (sending multiple concurrent requests at the same time, in a fault-tolerant way) Scrapy also gives you control over the politeness of the crawl through :ref:`a few settings <topics-settings-ref>`. You can do things like setting a download delay between each request, limiting the amount of concurrent requests per domain, and even :ref:`using an auto-throttling extension <topics-autothrottle>` that tries to figure these settings out automatically.

System Message: ERROR/3 (<stdin>, line 82); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 82); backlink

Unknown interpreted text role "ref".

Note

This is using :ref:`feed exports <topics-feed-exports>` to generate the JSON Lines file, you can easily change the export format (XML or CSV, for example) or the storage backend (FTP or Amazon S3, for example). You can also write an :ref:`item pipeline <topics-item-pipeline>` to store the items in a database.

System Message: ERROR/3 (<stdin>, line 92); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 92); backlink

Unknown interpreted text role "ref".

What else?

You've seen how to extract and store items from a website using Scrapy, but this is just the surface. Scrapy provides a lot of powerful features for making scraping easy and efficient, such as:

What's next?

The next steps for you are to :ref:`install Scrapy <intro-install>`, :ref:`follow through the tutorial <intro-tutorial>` to learn how to create a full-blown Scrapy project and join the community. Thanks for your interest!

System Message: ERROR/3 (<stdin>, line 149); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 149); backlink

Unknown interpreted text role "ref".
</html>