mirror of https://github.com/scrapy/scrapy.git
improved SCHEDULER_ORDER setting doc
--HG-- extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401067
This commit is contained in:
parent
512b75942c
commit
5f5962e6a4
|
|
@ -70,3 +70,8 @@ How can I simulate a user login in my spider?
|
|||
---------------------------------------------
|
||||
|
||||
See :ref:`ref-request-userlogin`.
|
||||
|
||||
Can I crawl in depth-first order instead of breadth-first order?
|
||||
----------------------------------------------------------------
|
||||
|
||||
Yes, there's a setting for that: :setting:`SCHEDULER_ORDER`.
|
||||
|
|
|
|||
|
|
@ -702,11 +702,23 @@ The scheduler to use for crawling.
|
|||
|
||||
.. setting:: SCHEDULER_ORDER
|
||||
|
||||
SCHEDULER_ORDER
|
||||
---------------
|
||||
|
||||
Default: ``'BFO'``
|
||||
|
||||
Scope: ``scrapy.core.scheduler``
|
||||
|
||||
The order to use for the crawling scheduler.
|
||||
The order to use for the crawling scheduler. Available orders are:
|
||||
|
||||
* ``'BFO'``: `Breadth-first order`_ - typically consumes more memory but
|
||||
reaches most relevant pages earlier.
|
||||
|
||||
* ``'DFO'``: `Depth-first order`_ - typically consumes less memory than
|
||||
but takes longer to reach most relevant pages.
|
||||
|
||||
.. _Breadth-first order: http://en.wikipedia.org/wiki/Breadth-first_search
|
||||
.. _Depth-first order: http://en.wikipedia.org/wiki/Depth-first_search
|
||||
|
||||
.. setting:: SCHEDULER_MIDDLEWARES
|
||||
|
||||
|
|
|
|||
|
|
@ -38,15 +38,6 @@ class Scheduler(object) :
|
|||
necesarily the order you put them in.
|
||||
|
||||
``pending_domains_count`` contains the names of all domains that are to be scheduled.
|
||||
|
||||
Two crawling orders are available by default, which can be set with the
|
||||
SCHEDULER_ORDER settings:
|
||||
|
||||
* BFO - breath-first order (default). Consumes more memory than DFO but reaches
|
||||
most relevant pages faster.
|
||||
|
||||
* DFO - depth-first order. Consumes less memory than BFO but usually takes
|
||||
longer to reach the most relevant pages.
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
|
|
|
|||
Loading…
Reference in New Issue