5.8 KiB
Optimizations
Scrapy offers different ways to optimize crawls based on :ref:`resource constraints <optimize-resources>` and :ref:`use cases <broad-crawls>`.
System Message: ERROR/3 (<stdin>, line 7); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 7); backlink
Unknown interpreted text role "ref".Lowering resource usage
…
Optimizing broad crawls
While Scrapy is well suited for broad crawls, i.e. crawls that target many websites, the default :ref:`settings <topics-settings>` are optimized for crawls targetting a single website.
System Message: ERROR/3 (<stdin>, line 29); backlink
Unknown interpreted text role "ref".For broad crawls, consider these adjustments:
Set :setting:`SCHEDULER_PRIORITY_QUEUE` to :class:`~scrapy.pqueues.DownloaderAwarePriorityQueue`.
System Message: ERROR/3 (<stdin>, line 37); backlink
Unknown interpreted text role "setting".
System Message: ERROR/3 (<stdin>, line 37); backlink
Unknown interpreted text role "class".
Increase concurrency
Concurrency is the number of requests that are processed in parallel. There is a global limit (:setting:`CONCURRENT_REQUESTS`) and an additional limit that can be set either per domain (:setting:`CONCURRENT_REQUESTS_PER_DOMAIN`) or per IP (:setting:`CONCURRENT_REQUESTS_PER_IP`).
System Message: ERROR/3 (<stdin>, line 45); backlink
Unknown interpreted text role "setting".System Message: ERROR/3 (<stdin>, line 45); backlink
Unknown interpreted text role "setting".System Message: ERROR/3 (<stdin>, line 45); backlink
Unknown interpreted text role "setting".Note
The scheduler priority queue :ref:`recommended for broad crawls <broad-crawls-scheduler-priority-queue>` does not support :setting:`CONCURRENT_REQUESTS_PER_IP`.
System Message: ERROR/3 (<stdin>, line 50); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 50); backlink
Unknown interpreted text role "setting".The default global concurrency limit in Scrapy is not suitable for crawling many different domains in parallel, so you will want to increase it. How much to increase it will depend on how much CPU and memory your crawler will have available.
A good starting point is 100:
System Message: WARNING/2 (<stdin>, line 61)
Cannot analyze code. Pygments package not found.
.. code-block:: python
CONCURRENT_REQUESTS = 100
But the best way to find out is by doing some trials and identifying at what concurrency your Scrapy process gets CPU bounded. For optimum performance, you should pick a concurrency where CPU usage is at 80-90%.
Increasing concurrency also increases memory usage. If memory usage is a concern, you might need to lower your global concurrency limit accordingly.
Increase Twisted IO thread pool maximum size
Currently Scrapy does DNS resolution in a blocking way with usage of thread pool. With higher concurrency levels the crawling could be slow or even fail hitting DNS resolver timeouts. Possible solution to increase the number of threads handling DNS queries. The DNS queue will be processed faster speeding up establishing of connection and crawling overall.
To increase maximum thread pool size use:
System Message: WARNING/2 (<stdin>, line 84)
Cannot analyze code. Pygments package not found.
.. code-block:: python
REACTOR_THREADPOOL_MAXSIZE = 20
Setup your own DNS
If you have multiple crawling processes and single central DNS, it can act like DoS attack on the DNS server resulting to slow down of entire network or even blocking your machines. To avoid this setup your own DNS server with local cache and upstream to some large DNS like OpenDNS or Verizon.
Reduce log level
When doing broad crawls you are often only interested in the crawl rates you get and any errors found. These stats are reported by Scrapy when using the INFO log level. In order to save CPU (and log storage requirements) you should not use DEBUG log level when performing large broad crawls in production. Using DEBUG level when developing your (broad) crawler may be fine though.
To set the log level use:
System Message: WARNING/2 (<stdin>, line 108)
Cannot analyze code. Pygments package not found.
.. code-block:: python
LOG_LEVEL = "INFO"
Disable retries
Retrying failed HTTP requests can slow down the crawls substantially, specially when sites causes are very slow (or fail) to respond, thus causing a timeout error which gets retried many times, unnecessarily, preventing crawler capacity to be reused for other domains.
To disable retries use:
System Message: WARNING/2 (<stdin>, line 136)
Cannot analyze code. Pygments package not found.
.. code-block:: python
RETRY_ENABLED = False
Reduce download timeout
Unless you are crawling from a very slow connection (which shouldn't be the case for broad crawls) reduce the download timeout so that stuck requests are discarded quickly and free up capacity to process the next ones.
To reduce the download timeout use:
System Message: WARNING/2 (<stdin>, line 149)
Cannot analyze code. Pygments package not found.
.. code-block:: python
DOWNLOAD_TIMEOUT = 15
Disable redirects
Consider disabling redirects, unless you are interested in following them. When doing broad crawls it's common to save redirects and resolve them when revisiting the site at a later crawl. This also help to keep the number of request constant per crawl batch, otherwise redirect loops may cause the crawler to dedicate too many resources on any specific domain.
To disable redirects use:
System Message: WARNING/2 (<stdin>, line 164)
Cannot analyze code. Pygments package not found.
.. code-block:: python
REDIRECT_ENABLED = False
Crawl in BFO order
:ref:`Scrapy crawls in DFO order by default <faq-bfo-dfo>`.
System Message: ERROR/3 (<stdin>, line 173); backlink
Unknown interpreted text role "ref".In broad crawls, however, page crawling tends to be faster than page processing. As a result, unprocessed early requests stay in memory until the final depth is reached, which can significantly increase memory usage.
:ref:`Crawl in BFO order <faq-bfo-dfo>` instead to save memory.
System Message: ERROR/3 (<stdin>, line 179); backlink
Unknown interpreted text role "ref".Be mindful of memory leaks
If your broad crawl shows a high memory usage, in addition to :ref:`crawling in BFO order <broad-crawls-bfo>` and :ref:`lowering concurrency <broad-crawls-concurrency>` you should :ref:`debug your memory leaks <topics-leaks>`.
System Message: ERROR/3 (<stdin>, line 185); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 185); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 185); backlink
Unknown interpreted text role "ref".Install a specific Twisted reactor
If the crawl is exceeding the system's capabilities, you might want to try installing a specific Twisted reactor, via the :setting:`TWISTED_REACTOR` setting.
System Message: ERROR/3 (<stdin>, line 194); backlink
Unknown interpreted text role "setting".