3.3 KiB
Scheduler
System Message: ERROR/3 (<stdin>, line 7)
Unknown directive type "module".
.. module:: scrapy.core.scheduler
The scheduler is a :ref:`component <topics-components>` that :ref:`stores pending requests, drops unwanted requests, and determines in which order pending requests are sent <scheduler-responsibilities>`.
System Message: ERROR/3 (<stdin>, line 9); backlink
Unknown interpreted text role "ref".System Message: ERROR/3 (<stdin>, line 9); backlink
Unknown interpreted text role "ref".It is set in the :setting:`SCHEDULER` setting. There is only 1 built-in scheduler, :class:`~scrapy.core.scheduler.Scheduler`, but you can :ref:`implement your own <custom-scheduler>`.
System Message: ERROR/3 (<stdin>, line 13); backlink
Unknown interpreted text role "setting".System Message: ERROR/3 (<stdin>, line 13); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 13); backlink
Unknown interpreted text role "ref".Scheduler responsibilities
A scheduler must:
Store pending requests.
The built-in scheduler stores requests in memory or disk. Other schedulers may rely, for example, on frontier, queue, database or storage services.
Pending requests may come from the :meth:`Spider.start <scrapy.Spider.start>` method, from spider callbacks (:attr:`Request.callback <scrapy.Request.callback>`), from :ref:`spider middlewares <topics-spider-middleware>` or from :ref:`downloader middlewares <topics-downloader-middleware>`.
System Message: ERROR/3 (<stdin>, line 29); backlink
Unknown interpreted text role "meth".
System Message: ERROR/3 (<stdin>, line 29); backlink
Unknown interpreted text role "attr".
System Message: ERROR/3 (<stdin>, line 29); backlink
Unknown interpreted text role "ref".
System Message: ERROR/3 (<stdin>, line 29); backlink
Unknown interpreted text role "ref".
Drop unwanted requests.
It is recommended for schedulers to use the configured :setting:`DUPEFILTER_CLASS` and take into account :attr:`Request.dont_filter <scrapy.Request.dont_filter>`, but schedulers may follow different criteria for dropping requests.
System Message: ERROR/3 (<stdin>, line 37); backlink
Unknown interpreted text role "setting".
System Message: ERROR/3 (<stdin>, line 37); backlink
Unknown interpreted text role "attr".
Return requests in the order they should be sent.
To determine the right order, schedulers may take into account :attr:`Request.priority <scrapy.Request.priority>` and applicable built-in settings (e.g. :setting:`SCHEDULER_PRIORITY_QUEUE`, :setting:`SCHEDULER_MEMORY_QUEUE`, :setting:`SCHEDULER_DISK_QUEUE`), but schedulers may also ignore any of those parameters at will.
System Message: ERROR/3 (<stdin>, line 44); backlink
Unknown interpreted text role "attr".
System Message: ERROR/3 (<stdin>, line 44); backlink
Unknown interpreted text role "setting".
System Message: ERROR/3 (<stdin>, line 44); backlink
Unknown interpreted text role "setting".
System Message: ERROR/3 (<stdin>, line 44); backlink
Unknown interpreted text role "setting".
Built-in components
Built-in scheduler
System Message: ERROR/3 (<stdin>, line 56)
Unknown directive type "autoclass".
.. autoclass:: Scheduler() :members: :special-members: __init__, __len__, pause, unpause
Built-in priority queues
Components for :setting:`SCHEDULER_PRIORITY_QUEUE`:
System Message: ERROR/3 (<stdin>, line 66); backlink
Unknown interpreted text role "setting".System Message: ERROR/3 (<stdin>, line 68)
Unknown directive type "autoclass".
.. autoclass:: scrapy.pqueues.ScrapyPriorityQueue()
System Message: ERROR/3 (<stdin>, line 69)
Unknown directive type "autoclass".
.. autoclass:: scrapy.pqueues.DownloaderAwarePriorityQueue()
Built-in memory queues
Components for :setting:`SCHEDULER_MEMORY_QUEUE`:
System Message: ERROR/3 (<stdin>, line 77); backlink
Unknown interpreted text role "setting".System Message: ERROR/3 (<stdin>, line 79)
Unknown directive type "autoclass".
.. autoclass:: scrapy.squeues.FifoMemoryQueue()
System Message: ERROR/3 (<stdin>, line 80)
Unknown directive type "autoclass".
.. autoclass:: scrapy.squeues.LifoMemoryQueue()
Built-in disk queues
Components for :setting:`SCHEDULER_DISK_QUEUE`:
System Message: ERROR/3 (<stdin>, line 88); backlink
Unknown interpreted text role "setting".System Message: ERROR/3 (<stdin>, line 90)
Unknown directive type "autoclass".
.. autoclass:: scrapy.squeues.PickleFifoDiskQueue()
System Message: ERROR/3 (<stdin>, line 91)
Unknown directive type "autoclass".
.. autoclass:: scrapy.squeues.PickleLifoDiskQueue()
System Message: ERROR/3 (<stdin>, line 92)
Unknown directive type "autoclass".
.. autoclass:: scrapy.squeues.MarshalFifoDiskQueue()
System Message: ERROR/3 (<stdin>, line 93)
Unknown directive type "autoclass".
.. autoclass:: scrapy.squeues.MarshalLifoDiskQueue()
Writing custom components
Writing a scheduler
Schedulers should subclass :class:`BaseScheduler` and implement its abstract methods:
System Message: ERROR/3 (<stdin>, line 104); backlink
Unknown interpreted text role "class".System Message: ERROR/3 (<stdin>, line 107)
Unknown directive type "autoclass".
.. autoclass:: BaseScheduler
:members:
:member-order: bysource
Writing a priority queue
System Message: ERROR/3 (<stdin>, line 117)
Unknown directive type "autoclass".
.. autoclass:: scrapy.pqueues.PriorityQueueProtocol()
:members:
:special-members: __init__, __len__
:member-order: bysource
Writing an internal queue
System Message: ERROR/3 (<stdin>, line 128)
Unknown directive type "autoclass".
.. autoclass:: scrapy.pqueues.QueueProtocol()
:members:
:special-members: __len__
:member-order: bysource