scrapy/docs/topics/optimize.rst

.. _optimize:

============
Optimization
============

Scrapy offers different ways to speed up crawls for specific :ref:`use cases
<broad-crawls>` and to :ref:`lower resource usage <optimize-resources>`.

.. _broad-crawls:
.. _topics-broad-crawls:

Speeding up broad crawls
========================

While Scrapy is well suited for **broad crawls**, i.e. crawls that target many
websites, the default :ref:`settings <topics-settings>` are optimized for
crawls targetting a single website.

For broad crawls, consider these adjustments:

-   .. _broad-crawls-concurrency:

    Increase the global concurrency:

    -   Set :setting:`CONCURRENT_REQUESTS` as close to
        :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` × [number of target domains]
        (e.g. 8 × 10 domains = 80 concurrent requests) as your CPU and memory
        allow.

    -   Increase :setting:`SCRAPER_SLOT_MAX_ACTIVE_SIZE` when increasing
        :setting:`CONCURRENT_REQUESTS` stops making a difference.

    If your CPU or memory become a bottleneck, see :ref:`optimize-resources`.

-   Optimize request scheduling:

    -   .. _broad-crawls-scheduler-priority-queue:

        Set :setting:`SCHEDULER_PRIORITY_QUEUE` to
        :class:`~scrapy.pqueues.DownloaderAwarePriorityQueue`.

    -   .. _broad-crawls-bfo:

        If memory is a bottleneck, see if :ref:`crawling in breadth-first order
        (BFO) <faq-bfo-dfo>` lowers memory usage.

-   Improve DNS resolution speed:

    -   Set up your own DNS server, with a local cache and upstream to a `large
        DNS server`_, to avoid slowing down your network.

        .. _large DNS server: https://en.wikipedia.org/wiki/Public_recursive_name_server#Notable_public_DNS_service_operators

    -   Increase :setting:`REACTOR_THREADPOOL_MAXSIZE` to the minimum value
        that avoids DNS resolution timeouts and makes a noticeable positive
        impact in crawl speed.

-   Lower the negative impact of some responses:

    -   Set :setting:`RETRY_ENABLED` to ``False`` or, if you need retries,
        consider lowering :setting:`RETRY_TIMES` and fine-tuning other
        :ref:`retry settings <retry-settings>`.

    -   Lower :setting:`DOWNLOAD_TIMEOUT` to a more reasonable value, to
        discard stuck requests more quickly.

    -   Set :setting:`REDIRECT_ENABLED` to ``False`` unless you want to follow
        redirects.


.. _optimize-resources:

Lowering resource usage
=======================

.. _optimize-memory:

Lowering memory usage
---------------------

-   Lower :setting:`SCRAPER_SLOT_MAX_ACTIVE_SIZE`.

-   Lower the number of :ref:`scheduled requests <topics-scheduler>` held in
    memory:

    -   Increase the :attr:`~scrapy.Request.priority` of requests whose
        :attr:`~scrapy.Request.callback` cannot yield additional
        requests.

        For example, the following spider uses a higher priority (1) for book
        requests than for pagination requests:

        .. code-block:: python

            from scrapy import Spider


            class BooksToScrapeComSpider(Spider):
                name = "books_toscrape_com"
                start_urls = [
                    "http://books.toscrape.com/catalogue/category/books/mystery_3/index.html"
                ]

                def parse(self, response):
                    next_page_links = response.css(".next a")
                    yield from response.follow_all(next_page_links)
                    book_links = response.css("article a")
                    yield from response.follow_all(book_links, callback=self.parse_book, priority=1)

                def parse_book(self, response):
                    yield {
                        "name": response.css("h1::text").get(),
                        "price": response.css(".price_color::text").re_first("£(.*)"),
                        "url": response.url,
                    }

        .. note:: If the number of request-yielding, low-priority requests
            scheduled at any given time is lower than concurrency settings
            (:setting:`CONCURRENT_REQUESTS_PER_DOMAIN` or
            :setting:`CONCURRENT_REQUESTS`), as in the example above, this can
            slow down your crawl by turning those requests into a bottleneck.

    -   If you have multiple :ref:`start requests <start-requests>`, consider
        :ref:`lazy <start-requests-lazy>` or :ref:`idle <start-requests-idle>`
        scheduling.

    -   Set :setting:`JOBDIR` to offload all scheduled requests to disk.

-   Be in the lookout for :ref:`memory leaks <topics-leaks>`.


Lowering network usage
----------------------

-   Install brotli_ and zstandard_ to support brotli-compressed_ and
    zstd-compressed_ responses.

    .. _brotli-compressed: https://www.ietf.org/rfc/rfc7932.txt
    .. _brotli: https://pypi.org/project/Brotli/
    .. _zstd-compressed: https://www.ietf.org/rfc/rfc8478.txt
    .. _zstandard: https://pypi.org/project/zstandard/


Lowering CPU usage
------------------

-   Set :setting:`LOG_LEVEL` to ``"INFO"`` or higher.


Other tips
----------

-   Try :ref:`using the asyncio reactor <install-asyncio>`, installing
    :doc:`uvloop <uvloop:index>` and setting :setting:`ASYNCIO_EVENT_LOOP` to
    :class:`uvloop.Loop`.

    Alternatively, try switching :setting:`TWISTED_REACTOR` to :doc:`some other
    reactor <core/howto/choosing-reactor>`.

-   Disable unused :ref:`components <topics-components>`.

    For example, set :setting:`COOKIES_ENABLED` to ``False`` unless you need
    cookies.