RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Add documentation 

add section to broad-crawl topic
reword in accord with broad-crawl topic
add documentation for new priority queue
This commit is contained in:
Vostretsov Nikita 2018-12-25 09:44:20 +00:00
parent 8e8ce301b1
commit 338b78d796
2 changed files with 17 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
docs/topics/broad-crawls.rst
View File

@ -39,6 +39,17 @@ you need to keep in mind when using Scrapy for doing broad crawls, along with
concrete suggestions of Scrapy settings to tune in order to achieve an
efficient broad crawl.
Use proper :setting:`SCHEDULER_PRIORITY_QUEUE`
==============================================
Default scrapy's scheduler priority queue is ``'queuelib.PriorityQueue'``.
It works best during single domain crawl. And it does not work well with crawling
many different domains in parallel
To apply recommended priority queue use::
SCHEDULER_PRIORITY_QUEUE = 'scrapy.pqueues.DownloaderAwarePriorityQueue'
Increase concurrency
====================

7
docs/topics/settings.rst
View File

@ -1144,7 +1144,12 @@ SCHEDULER_PRIORITY_QUEUE
------------------------
Default: ``'queuelib.PriorityQueue'``
Type of priority queue used by scheduler.
Type of priority queue used by scheduler. Another available type is
``scrapy.pqueues.DownloaderAwarePriorityQueue``.
``scrapy.pqueues.DownloaderAwarePriorityQueue`` is works better than
``'queuelib.PriorityQueue'`` when you crawl many different domains in parallel.
But ``scrapy.pqueues.DownloaderAwarePriorityQueue``
does not work together with :setting:`CONCURRENT_REQUESTS_PER_IP`.
.. setting:: SPIDER_CONTRACTS