From 783de8a8d8c6c3c0a91318c3354034a7eb46a627 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A1n=20Chaves?= Date: Tue, 12 Feb 2019 16:36:35 +0100 Subject: [PATCH] Document how request concurrency settings impact request order --- docs/faq.rst | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/docs/faq.rst b/docs/faq.rst index 7a0628f88..f6dc9cec0 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -129,13 +129,23 @@ Does Scrapy crawl in breadth-first or depth-first order? By default, Scrapy uses a `LIFO`_ queue for storing pending requests, which basically means that it crawls in `DFO order`_. This order is more convenient -in most cases. If you do want to crawl in true `BFO order`_, you can do it by +in most cases. + +If you do want to crawl in true `BFO order`_, you can do it by setting the following settings:: DEPTH_PRIORITY = 1 SCHEDULER_DISK_QUEUE = 'scrapy.squeues.PickleFifoDiskQueue' SCHEDULER_MEMORY_QUEUE = 'scrapy.squeues.FifoMemoryQueue' +While pending requests are below the configured values of +:setting:`CONCURRENT_REQUESTS`, :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` or +:setting:`CONCURRENT_REQUESTS_PER_DOMAIN`, those requests are sent +concurrently. As a result, the first few requests of a crawl rarely follow the +desired order. Lowering those settings to ``1`` enforces the desired order, but +it significantly slows down the crawl as a whole. + + My Scrapy crawler has memory leaks. What can I do? --------------------------------------------------