RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
scrapy/docs/topics/optimize.rst

5.4 KiB
Raw Permalink Blame History

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> </head>

Optimization

Scrapy offers different ways to speed up crawls for specific :ref:`use cases <broad-crawls>` and to :ref:`lower resource usage <optimize-resources>`.

System Message: ERROR/3 (<stdin>, line 7); backlink

Unknown interpreted text role "ref".

System Message: ERROR/3 (<stdin>, line 7); backlink

Unknown interpreted text role "ref".

Speeding up broad crawls

While Scrapy is well suited for broad crawls, i.e. crawls that target many websites, the default :ref:`settings <topics-settings>` are optimized for crawls targetting a single website.

System Message: ERROR/3 (<stdin>, line 16); backlink

Unknown interpreted text role "ref".

For broad crawls, consider these adjustments:

Lowering resource usage

Lowering memory usage

  • Lower :setting:`SCRAPER_SLOT_MAX_ACTIVE_SIZE`.

    System Message: ERROR/3 (<stdin>, line 82); backlink

    Unknown interpreted text role "setting".

  • Lower the number of :ref:`scheduled requests <topics-scheduler>` held in memory:

    System Message: ERROR/3 (<stdin>, line 84); backlink

    Unknown interpreted text role "ref".

    • Increase the :attr:`~scrapy.Request.priority` of requests whose :attr:`~scrapy.Request.callback` cannot yield additional requests.

      System Message: ERROR/3 (<stdin>, line 87); backlink

      Unknown interpreted text role "attr".

      System Message: ERROR/3 (<stdin>, line 87); backlink

      Unknown interpreted text role "attr".

      For example, the following spider uses a higher priority (1) for book requests than for pagination requests:

      System Message: WARNING/2 (<stdin>, line 94)

      Cannot analyze code. Pygments package not found.

      .. code-block:: python

    from scrapy import Spider


    class BooksToScrapeComSpider(Spider):
        name = "books_toscrape_com"
        start_urls = [
            "http://books.toscrape.com/catalogue/category/books/mystery_3/index.html"
        ]

        def parse(self, response):
            next_page_links = response.css(".next a")
            yield from response.follow_all(next_page_links)
            book_links = response.css("article a")
            yield from response.follow_all(book_links, callback=self.parse_book, priority=1)

        def parse_book(self, response):
            yield {
                "name": response.css("h1::text").get(),
                "price": response.css(".price_color::text").re_first("£(.*)"),
                "url": response.url,
            }

      Note

      If the number of request-yielding, low-priority requests scheduled at any given time is lower than concurrency settings (:setting:`CONCURRENT_REQUESTS_PER_DOMAIN` or :setting:`CONCURRENT_REQUESTS`), as in the example above, this can slow down your crawl by turning those requests into a bottleneck.

      System Message: ERROR/3 (<stdin>, line 118); backlink

      Unknown interpreted text role "setting".

      System Message: ERROR/3 (<stdin>, line 118); backlink

      Unknown interpreted text role "setting".

    • If you have multiple :ref:`start requests <start-requests>`, consider :ref:`lazy <start-requests-lazy>` or :ref:`idle <start-requests-idle>` scheduling.

      System Message: ERROR/3 (<stdin>, line 124); backlink

      Unknown interpreted text role "ref".

      System Message: ERROR/3 (<stdin>, line 124); backlink

      Unknown interpreted text role "ref".

      System Message: ERROR/3 (<stdin>, line 124); backlink

      Unknown interpreted text role "ref".

    • Set :setting:`JOBDIR` to offload all scheduled requests to disk.

      System Message: ERROR/3 (<stdin>, line 128); backlink

      Unknown interpreted text role "setting".

  • Be in the lookout for :ref:`memory leaks <topics-leaks>`.

    System Message: ERROR/3 (<stdin>, line 130); backlink

    Unknown interpreted text role "ref".

Lowering network usage

Lowering CPU usage

  • Set :setting:`LOG_LEVEL` to "INFO" or higher.

    System Message: ERROR/3 (<stdin>, line 148); backlink

    Unknown interpreted text role "setting".

Other tips

</html>