3.9 KiB
Optimization
Scrapy offers different ways to speed up crawls for specific :ref:`use cases <broad-crawls>` and to :ref:`lower resource usage <optimize-resources>`.
System Message: ERROR/3 (<stdin>, line 7); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 7); backlink
Unknown interpreted text role "ref".Speeding up broad crawls
While Scrapy is well suited for broad crawls, i.e. crawls that target many websites, the default :ref:`settings <topics-settings>` are optimized for crawls targetting a single website.
System Message: ERROR/3 (<stdin>, line 16); backlink
Unknown interpreted text role "ref".For broad crawls, consider these adjustments:
Increase global concurrency:
Set :setting:`CONCURRENT_REQUESTS` as close to “number of target domains” times :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` (e.g. 80 for 10 domains) as your CPU and memory allow.
System Message: ERROR/3 (<stdin>, line 26); backlink
Unknown interpreted text role "setting".
System Message: ERROR/3 (<stdin>, line 26); backlink
Unknown interpreted text role "setting".
Increase :setting:`SCRAPER_SLOT_MAX_ACTIVE_SIZE` when increasing :setting:`CONCURRENT_REQUESTS` stops making a difference.
System Message: ERROR/3 (<stdin>, line 30); backlink
Unknown interpreted text role "setting".
System Message: ERROR/3 (<stdin>, line 30); backlink
Unknown interpreted text role "setting".
If your CPU or memory become a bottleneck, see :ref:`optimize-resources`.
System Message: ERROR/3 (<stdin>, line 33); backlink
Unknown interpreted text role "ref".
Optimize request scheduling:
Set :setting:`SCHEDULER_PRIORITY_QUEUE` to :class:`~scrapy.pqueues.DownloaderAwarePriorityQueue`.
System Message: ERROR/3 (<stdin>, line 39); backlink
Unknown interpreted text role "setting".
System Message: ERROR/3 (<stdin>, line 39); backlink
Unknown interpreted text role "class".
If memory is a bottleneck, see if :ref:`crawling in breadth-first order (BFO) <faq-bfo-dfo>` lowers memory usage.
System Message: ERROR/3 (<stdin>, line 44); backlink
Unknown interpreted text role "ref".
Improve DNS resolution speed:
Set up your own DNS server, with a local cache and upstream to a large DNS server, to avoid slowing down your network.
Increase :setting:`REACTOR_THREADPOOL_MAXSIZE` to the minimum value that avoids DNS resolution timeouts and makes a noticeable positive impact in crawl speed.
System Message: ERROR/3 (<stdin>, line 54); backlink
Unknown interpreted text role "setting".
Lower the negative impact of some responses:
Set :setting:`RETRY_ENABLED` to False or, if you need retries, consider lowering :setting:`RETRY_TIMES` and fine-tuning other :ref:`retry settings <retry-settings>`.
System Message: ERROR/3 (<stdin>, line 60); backlink
Unknown interpreted text role "setting".
System Message: ERROR/3 (<stdin>, line 60); backlink
Unknown interpreted text role "setting".
System Message: ERROR/3 (<stdin>, line 60); backlink
Unknown interpreted text role "ref".
Lower :setting:`DOWNLOAD_TIMEOUT` to a more reasonable value, to discard stuck requests more quickly.
System Message: ERROR/3 (<stdin>, line 64); backlink
Unknown interpreted text role "setting".
Set :setting:`REDIRECT_ENABLED` to False unless you want to follow redirects.
System Message: ERROR/3 (<stdin>, line 67); backlink
Unknown interpreted text role "setting".
Lowering resource usage
Lowering memory usage
Lower :setting:`SCRAPER_SLOT_MAX_ACTIVE_SIZE`.
System Message: ERROR/3 (<stdin>, line 79); backlink
Unknown interpreted text role "setting".
Lower the number of :ref:`scheduled requests <topics-scheduler>` held in memory:
System Message: ERROR/3 (<stdin>, line 81); backlink
Unknown interpreted text role "ref".
Increase the :attr:`~scrapy.Request.priority` of requests whose :attr:`~scrapy.Request.callback` cannot yield additional requests.
System Message: ERROR/3 (<stdin>, line 84); backlink
Unknown interpreted text role "attr".
System Message: ERROR/3 (<stdin>, line 84); backlink
Unknown interpreted text role "attr".
If you have many :ref:`start requests <start-requests>`, use :ref:`lazy scheduling <start-requests-lazy>`.
System Message: ERROR/3 (<stdin>, line 88); backlink
Unknown interpreted text role "ref".
System Message: ERROR/3 (<stdin>, line 88); backlink
Unknown interpreted text role "ref".
Set :setting:`JOBDIR` to offload all scheduled requests to disk.
System Message: ERROR/3 (<stdin>, line 91); backlink
Unknown interpreted text role "setting".
Be in the lookout for :ref:`memory leaks <topics-leaks>`.
System Message: ERROR/3 (<stdin>, line 93); backlink
Unknown interpreted text role "ref".
Lowering network usage
- Install brotli and zstandard to support brotli-compressed and zstd-compressed responses.
Lowering CPU usage
Set :setting:`LOG_LEVEL` to "INFO" or higher.
System Message: ERROR/3 (<stdin>, line 111); backlink
Unknown interpreted text role "setting".
Other tips
Try :ref:`using the asyncio reactor <install-asyncio>`, installing :doc:`uvloop <uvloop:index>` and setting :setting:`ASYNCIO_EVENT_LOOP` to :class:`uvloop.Loop`.
System Message: ERROR/3 (<stdin>, line 117); backlink
Unknown interpreted text role "ref".
System Message: ERROR/3 (<stdin>, line 117); backlink
Unknown interpreted text role "doc".
System Message: ERROR/3 (<stdin>, line 117); backlink
Unknown interpreted text role "setting".
System Message: ERROR/3 (<stdin>, line 117); backlink
Unknown interpreted text role "class".
Alternatively, try switching :setting:`TWISTED_REACTOR` to :doc:`some other reactor <core/howto/choosing-reactor>`.
System Message: ERROR/3 (<stdin>, line 121); backlink
Unknown interpreted text role "setting".
System Message: ERROR/3 (<stdin>, line 121); backlink
Unknown interpreted text role "doc".
Disable unused :ref:`components <topics-components>`.
System Message: ERROR/3 (<stdin>, line 124); backlink
Unknown interpreted text role "ref".
For example, set :setting:`COOKIES_ENABLED` to False unless you need cookies.
System Message: ERROR/3 (<stdin>, line 126); backlink
Unknown interpreted text role "setting".