4.0 KiB
Optimization
Scrapy offers different ways to speed up crawls for specific :ref:`use cases <broad-crawls>` and to :ref:`lower resource usage <optimize-resources>`.
System Message: ERROR/3 (<stdin>, line 7); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 7); backlink
Unknown interpreted text role "ref".Speeding up broad crawls
While Scrapy is well suited for broad crawls, i.e. crawls that target many websites, the default :ref:`settings <topics-settings>` are optimized for crawls targetting a single website.
System Message: ERROR/3 (<stdin>, line 16); backlink
Unknown interpreted text role "ref".For broad crawls, consider these adjustments:
Increase the global concurrency:
Set :setting:`CONCURRENT_REQUESTS` as close to :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` × [number of target domains] (e.g. 8 × 10 domains = 80 concurrent requests) as your CPU and memory allow.
System Message: ERROR/3 (<stdin>, line 26); backlink
Unknown interpreted text role "setting".
System Message: ERROR/3 (<stdin>, line 26); backlink
Unknown interpreted text role "setting".
Increase :setting:`SCRAPER_SLOT_MAX_ACTIVE_SIZE` when increasing :setting:`CONCURRENT_REQUESTS` stops making a difference.
System Message: ERROR/3 (<stdin>, line 31); backlink
Unknown interpreted text role "setting".
System Message: ERROR/3 (<stdin>, line 31); backlink
Unknown interpreted text role "setting".
If your CPU or memory become a bottleneck, see :ref:`optimize-resources`.
System Message: ERROR/3 (<stdin>, line 34); backlink
Unknown interpreted text role "ref".
Optimize request scheduling:
Set :setting:`SCHEDULER_PRIORITY_QUEUE` to :class:`~scrapy.pqueues.DownloaderAwarePriorityQueue`.
System Message: ERROR/3 (<stdin>, line 40); backlink
Unknown interpreted text role "setting".
System Message: ERROR/3 (<stdin>, line 40); backlink
Unknown interpreted text role "class".
If memory is a bottleneck, see if :ref:`crawling in breadth-first order (BFO) <faq-bfo-dfo>` lowers memory usage.
System Message: ERROR/3 (<stdin>, line 45); backlink
Unknown interpreted text role "ref".
Improve DNS resolution speed:
Set up your own DNS server, with a local cache and upstream to a large DNS server, to avoid slowing down your network.
Increase :setting:`REACTOR_THREADPOOL_MAXSIZE` to the minimum value that avoids DNS resolution timeouts and makes a noticeable positive impact in crawl speed.
System Message: ERROR/3 (<stdin>, line 55); backlink
Unknown interpreted text role "setting".
Lower the negative impact of some responses:
Set :setting:`RETRY_ENABLED` to False or, if you need retries, consider lowering :setting:`RETRY_TIMES` and fine-tuning other :ref:`retry settings <retry-settings>`.
System Message: ERROR/3 (<stdin>, line 61); backlink
Unknown interpreted text role "setting".
System Message: ERROR/3 (<stdin>, line 61); backlink
Unknown interpreted text role "setting".
System Message: ERROR/3 (<stdin>, line 61); backlink
Unknown interpreted text role "ref".
Lower :setting:`DOWNLOAD_TIMEOUT` to a more reasonable value, to discard stuck requests more quickly.
System Message: ERROR/3 (<stdin>, line 65); backlink
Unknown interpreted text role "setting".
Set :setting:`REDIRECT_ENABLED` to False unless you want to follow redirects.
System Message: ERROR/3 (<stdin>, line 68); backlink
Unknown interpreted text role "setting".
Lowering resource usage
Lowering memory usage
Lower :setting:`SCRAPER_SLOT_MAX_ACTIVE_SIZE`.
System Message: ERROR/3 (<stdin>, line 82); backlink
Unknown interpreted text role "setting".
Lower the number of :ref:`scheduled requests <topics-scheduler>` held in memory:
System Message: ERROR/3 (<stdin>, line 84); backlink
Unknown interpreted text role "ref".
Increase the :attr:`~scrapy.Request.priority` of requests whose :attr:`~scrapy.Request.callback` cannot yield additional requests.
System Message: ERROR/3 (<stdin>, line 87); backlink
Unknown interpreted text role "attr".
System Message: ERROR/3 (<stdin>, line 87); backlink
Unknown interpreted text role "attr".
If you have multiple :ref:`start requests <start-requests>`, consider :ref:`lazy <start-requests-lazy>` or :ref:`idle <start-requests-idle>` scheduling.
System Message: ERROR/3 (<stdin>, line 91); backlink
Unknown interpreted text role "ref".
System Message: ERROR/3 (<stdin>, line 91); backlink
Unknown interpreted text role "ref".
System Message: ERROR/3 (<stdin>, line 91); backlink
Unknown interpreted text role "ref".
Set :setting:`JOBDIR` to offload all scheduled requests to disk.
System Message: ERROR/3 (<stdin>, line 95); backlink
Unknown interpreted text role "setting".
Be in the lookout for :ref:`memory leaks <topics-leaks>`.
System Message: ERROR/3 (<stdin>, line 97); backlink
Unknown interpreted text role "ref".
Lowering network usage
- Install brotli and zstandard to support brotli-compressed and zstd-compressed responses.
Lowering CPU usage
Set :setting:`LOG_LEVEL` to "INFO" or higher.
System Message: ERROR/3 (<stdin>, line 115); backlink
Unknown interpreted text role "setting".
Other tips
Try :ref:`using the asyncio reactor <install-asyncio>`, installing :doc:`uvloop <uvloop:index>` and setting :setting:`ASYNCIO_EVENT_LOOP` to :class:`uvloop.Loop`.
System Message: ERROR/3 (<stdin>, line 121); backlink
Unknown interpreted text role "ref".
System Message: ERROR/3 (<stdin>, line 121); backlink
Unknown interpreted text role "doc".
System Message: ERROR/3 (<stdin>, line 121); backlink
Unknown interpreted text role "setting".
System Message: ERROR/3 (<stdin>, line 121); backlink
Unknown interpreted text role "class".
Alternatively, try switching :setting:`TWISTED_REACTOR` to :doc:`some other reactor <core/howto/choosing-reactor>`.
System Message: ERROR/3 (<stdin>, line 125); backlink
Unknown interpreted text role "setting".
System Message: ERROR/3 (<stdin>, line 125); backlink
Unknown interpreted text role "doc".
Disable unused :ref:`components <topics-components>`.
System Message: ERROR/3 (<stdin>, line 128); backlink
Unknown interpreted text role "ref".
For example, set :setting:`COOKIES_ENABLED` to False unless you need cookies.
System Message: ERROR/3 (<stdin>, line 130); backlink
Unknown interpreted text role "setting".