improved SCHEDULER_ORDER setting doc

--HG--
extra : convert_revision : svn%3Ab85faa78-f9eb-468e-a121-7cced6da292c%401067
This commit is contained in:
Pablo Hoffman 2009-04-17 13:06:31 +00:00
parent 512b75942c
commit 5f5962e6a4
3 changed files with 18 additions and 10 deletions

View File

@ -70,3 +70,8 @@ How can I simulate a user login in my spider?
---------------------------------------------
See :ref:`ref-request-userlogin`.
Can I crawl in depth-first order instead of breadth-first order?
----------------------------------------------------------------
Yes, there's a setting for that: :setting:`SCHEDULER_ORDER`.

View File

@ -702,11 +702,23 @@ The scheduler to use for crawling.
.. setting:: SCHEDULER_ORDER
SCHEDULER_ORDER
---------------
Default: ``'BFO'``
Scope: ``scrapy.core.scheduler``
The order to use for the crawling scheduler.
The order to use for the crawling scheduler. Available orders are:
* ``'BFO'``: `Breadth-first order`_ - typically consumes more memory but
reaches most relevant pages earlier.
* ``'DFO'``: `Depth-first order`_ - typically consumes less memory than
but takes longer to reach most relevant pages.
.. _Breadth-first order: http://en.wikipedia.org/wiki/Breadth-first_search
.. _Depth-first order: http://en.wikipedia.org/wiki/Depth-first_search
.. setting:: SCHEDULER_MIDDLEWARES

View File

@ -38,15 +38,6 @@ class Scheduler(object) :
necesarily the order you put them in.
``pending_domains_count`` contains the names of all domains that are to be scheduled.
Two crawling orders are available by default, which can be set with the
SCHEDULER_ORDER settings:
* BFO - breath-first order (default). Consumes more memory than DFO but reaches
most relevant pages faster.
* DFO - depth-first order. Consumes less memory than BFO but usually takes
longer to reach the most relevant pages.
"""
def __init__(self):