diff --git a/docs/topics/broad-crawls.rst b/docs/topics/broad-crawls.rst index eb02086dc..37f7a8748 100644 --- a/docs/topics/broad-crawls.rst +++ b/docs/topics/broad-crawls.rst @@ -39,6 +39,17 @@ you need to keep in mind when using Scrapy for doing broad crawls, along with concrete suggestions of Scrapy settings to tune in order to achieve an efficient broad crawl. +Use proper :setting:`SCHEDULER_PRIORITY_QUEUE` +============================================== + +Default scrapy's scheduler priority queue is ``'queuelib.PriorityQueue'``. +It works best during single domain crawl. And it does not work well with crawling +many different domains in parallel + +To apply recommended priority queue use:: + + SCHEDULER_PRIORITY_QUEUE = 'scrapy.pqueues.DownloaderAwarePriorityQueue' + Increase concurrency ==================== diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 47b6cf13d..7b9ff7e39 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -1144,7 +1144,12 @@ SCHEDULER_PRIORITY_QUEUE ------------------------ Default: ``'queuelib.PriorityQueue'`` -Type of priority queue used by scheduler. +Type of priority queue used by scheduler. Another available type is +``scrapy.pqueues.DownloaderAwarePriorityQueue``. +``scrapy.pqueues.DownloaderAwarePriorityQueue`` is works better than +``'queuelib.PriorityQueue'`` when you crawl many different domains in parallel. +But ``scrapy.pqueues.DownloaderAwarePriorityQueue`` +does not work together with :setting:`CONCURRENT_REQUESTS_PER_IP`. .. setting:: SPIDER_CONTRACTS