From 845bae6637239c859c9952c23f42902e36d10f6b Mon Sep 17 00:00:00 2001 From: Mikhail Korobov Date: Wed, 27 Mar 2019 08:49:19 +0000 Subject: [PATCH 1/5] Update docs/topics/broad-crawls.rst Co-Authored-By: whalebot-helmsman --- docs/topics/broad-crawls.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/topics/broad-crawls.rst b/docs/topics/broad-crawls.rst index 37f7a8748..64c8883b1 100644 --- a/docs/topics/broad-crawls.rst +++ b/docs/topics/broad-crawls.rst @@ -42,7 +42,7 @@ efficient broad crawl. Use proper :setting:`SCHEDULER_PRIORITY_QUEUE` ============================================== -Default scrapy's scheduler priority queue is ``'queuelib.PriorityQueue'``. +Default scrapy's scheduler priority queue is ``'scrapy.pqueues.ScrapyPriorityQueue'``. It works best during single domain crawl. And it does not work well with crawling many different domains in parallel From 46b9ab0c58354deb1045c20f3bc061526d69f356 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A1n=20Chaves?= Date: Fri, 29 Mar 2019 10:28:36 +0000 Subject: [PATCH 2/5] Update docs/topics/broad-crawls.rst Co-Authored-By: whalebot-helmsman --- docs/topics/broad-crawls.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/topics/broad-crawls.rst b/docs/topics/broad-crawls.rst index 64c8883b1..68a24a4d2 100644 --- a/docs/topics/broad-crawls.rst +++ b/docs/topics/broad-crawls.rst @@ -42,7 +42,7 @@ efficient broad crawl. Use proper :setting:`SCHEDULER_PRIORITY_QUEUE` ============================================== -Default scrapy's scheduler priority queue is ``'scrapy.pqueues.ScrapyPriorityQueue'``. +Scrapy’s default scheduler priority queue is ``'scrapy.pqueues.ScrapyPriorityQueue'``. It works best during single domain crawl. And it does not work well with crawling many different domains in parallel From e3df6be360a58f016e31d5bfa2e04cd2e5d1965b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A1n=20Chaves?= Date: Fri, 29 Mar 2019 10:28:52 +0000 Subject: [PATCH 3/5] Update docs/topics/broad-crawls.rst Co-Authored-By: whalebot-helmsman --- docs/topics/broad-crawls.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/topics/broad-crawls.rst b/docs/topics/broad-crawls.rst index 68a24a4d2..b149d7f4a 100644 --- a/docs/topics/broad-crawls.rst +++ b/docs/topics/broad-crawls.rst @@ -43,7 +43,7 @@ Use proper :setting:`SCHEDULER_PRIORITY_QUEUE` ============================================== Scrapy’s default scheduler priority queue is ``'scrapy.pqueues.ScrapyPriorityQueue'``. -It works best during single domain crawl. And it does not work well with crawling +It works best during single-domain crawl. It does not work well with crawling many different domains in parallel To apply recommended priority queue use:: From bd228f1d962c7f4759536d8cda278857de7d5234 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A1n=20Chaves?= Date: Fri, 29 Mar 2019 10:29:04 +0000 Subject: [PATCH 4/5] Update docs/topics/broad-crawls.rst Co-Authored-By: whalebot-helmsman --- docs/topics/broad-crawls.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/topics/broad-crawls.rst b/docs/topics/broad-crawls.rst index b149d7f4a..a01f28248 100644 --- a/docs/topics/broad-crawls.rst +++ b/docs/topics/broad-crawls.rst @@ -46,7 +46,7 @@ Scrapy’s default scheduler priority queue is ``'scrapy.pqueues.ScrapyPriorityQ It works best during single-domain crawl. It does not work well with crawling many different domains in parallel -To apply recommended priority queue use:: +To apply the recommended priority queue use:: SCHEDULER_PRIORITY_QUEUE = 'scrapy.pqueues.DownloaderAwarePriorityQueue' From 1ee99e1f4240af6a7a72fe7c58b89d7bce1cd09e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A1n=20Chaves?= Date: Fri, 29 Mar 2019 10:29:15 +0000 Subject: [PATCH 5/5] Update docs/topics/settings.rst Co-Authored-By: whalebot-helmsman --- docs/topics/settings.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index ed94146f4..4a5439bfc 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -1157,7 +1157,7 @@ SCHEDULER_PRIORITY_QUEUE ------------------------ Default: ``'scrapy.pqueues.ScrapyPriorityQueue'`` -Type of priority queue used by scheduler. Another available type is +Type of priority queue used by the scheduler. Another available type is ``scrapy.pqueues.DownloaderAwarePriorityQueue``. ``scrapy.pqueues.DownloaderAwarePriorityQueue`` works better than ``scrapy.pqueues.ScrapyPriorityQueue`` when you crawl many different