diff --git a/docs/faq.rst b/docs/faq.rst index 69edb0993..7105baeef 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -129,13 +129,23 @@ Does Scrapy crawl in breadth-first or depth-first order? By default, Scrapy uses a `LIFO`_ queue for storing pending requests, which basically means that it crawls in `DFO order`_. This order is more convenient -in most cases. If you do want to crawl in true `BFO order`_, you can do it by +in most cases. + +If you do want to crawl in true `BFO order`_, you can do it by setting the following settings:: DEPTH_PRIORITY = 1 SCHEDULER_DISK_QUEUE = 'scrapy.squeues.PickleFifoDiskQueue' SCHEDULER_MEMORY_QUEUE = 'scrapy.squeues.FifoMemoryQueue' +While pending requests are below the configured values of +:setting:`CONCURRENT_REQUESTS`, :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` or +:setting:`CONCURRENT_REQUESTS_PER_DOMAIN`, those requests are sent +concurrently. As a result, the first few requests of a crawl rarely follow the +desired order. Lowering those settings to ``1`` enforces the desired order, but +it significantly slows down the crawl as a whole. + + My Scrapy crawler has memory leaks. What can I do? --------------------------------------------------