RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'round-robin-scheduler-tested' of github.com:whalebot-helmsman/scrapy into round-robin-scheduler-tested

This commit is contained in:
Vostretsov Nikita 2019-03-29 10:31:52 +00:00
parent 554d872822 1ee99e1f42
commit 539a08cb85
2 changed files with 4 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
docs/topics/broad-crawls.rst
View File

@ -42,11 +42,11 @@ efficient broad crawl.
Use proper :setting:`SCHEDULER_PRIORITY_QUEUE`
==============================================
Default scrapy's scheduler priority queue is ``'queuelib.PriorityQueue'``.
It works best during single domain crawl. And it does not work well with crawling
Scrapy’s default scheduler priority queue is ``'scrapy.pqueues.ScrapyPriorityQueue'``.
It works best during single-domain crawl. It does not work well with crawling
many different domains in parallel
To apply recommended priority queue use::
To apply the recommended priority queue use::
SCHEDULER_PRIORITY_QUEUE = 'scrapy.pqueues.DownloaderAwarePriorityQueue'

2
docs/topics/settings.rst
View File

@ -1157,7 +1157,7 @@ SCHEDULER_PRIORITY_QUEUE
------------------------
Default: ``'scrapy.pqueues.ScrapyPriorityQueue'``
Type of priority queue used by scheduler. Another available type is
Type of priority queue used by the scheduler. Another available type is
``scrapy.pqueues.DownloaderAwarePriorityQueue``.
``scrapy.pqueues.DownloaderAwarePriorityQueue`` works better than
``scrapy.pqueues.ScrapyPriorityQueue`` when you crawl many different