mirror of https://github.com/scrapy/scrapy.git
Minor edits around Optimization
This commit is contained in:
parent
a858ddfef9
commit
28df50ddcc
|
|
@ -21,11 +21,12 @@ For broad crawls, consider these adjustments:
|
|||
|
||||
- .. _broad-crawls-concurrency:
|
||||
|
||||
Increase global concurrency:
|
||||
Increase the global concurrency:
|
||||
|
||||
- Set :setting:`CONCURRENT_REQUESTS` as close to “number of target
|
||||
domains” times :setting:`CONCURRENT_REQUESTS_PER_DOMAIN` (e.g. 80 for
|
||||
10 domains) as your CPU and memory allow.
|
||||
- Set :setting:`CONCURRENT_REQUESTS` as close to
|
||||
:setting:`CONCURRENT_REQUESTS_PER_DOMAIN` × [number of target domains]
|
||||
(e.g. 8 × 10 domains = 80 concurrent requests) as your CPU and memory
|
||||
allow.
|
||||
|
||||
- Increase :setting:`SCRAPER_SLOT_MAX_ACTIVE_SIZE` when increasing
|
||||
:setting:`CONCURRENT_REQUESTS` stops making a difference.
|
||||
|
|
@ -87,8 +88,9 @@ Lowering memory usage
|
|||
:attr:`~scrapy.Request.callback` cannot yield additional
|
||||
requests.
|
||||
|
||||
- If you have many :ref:`start requests <start-requests>`, use :ref:`lazy
|
||||
<start-requests-lazy>` or :ref:`idle <start-requests-idle>` scheduling.
|
||||
- If you have multiple :ref:`start requests <start-requests>`, consider
|
||||
:ref:`lazy <start-requests-lazy>` or :ref:`idle <start-requests-idle>`
|
||||
scheduling.
|
||||
|
||||
- Set :setting:`JOBDIR` to offload all scheduled requests to disk.
|
||||
|
||||
|
|
|
|||
|
|
@ -20,6 +20,7 @@ functionality not required in the base classes. These are described
|
|||
below in :ref:`topics-request-response-ref-request-subclasses` and
|
||||
:ref:`topics-request-response-ref-response-subclasses`.
|
||||
|
||||
.. _request:
|
||||
|
||||
Request objects
|
||||
===============
|
||||
|
|
|
|||
|
|
@ -369,13 +369,16 @@ See `Scrapyd documentation`_.
|
|||
Start requests
|
||||
==============
|
||||
|
||||
**Start requests** are the requests yielded by the :meth:`~scrapy.Spider.start`
|
||||
method of a spider.
|
||||
**Start requests** are the :ref:`requests <request>` that serve as a starting
|
||||
point for a spider.
|
||||
|
||||
By default, Scrapy does not try to send :meth:`~scrapy.Spider.start` requests
|
||||
in order. Instead, it prioritizes reaching :setting:`CONCURRENT_REQUESTS` and
|
||||
:ref:`scheduling <topics-scheduler>` start requests. However, you can change
|
||||
that.
|
||||
They are yielded by the :meth:`~scrapy.Spider.start` method, and may be
|
||||
modified by :ref:`spider middlewares <topics-spider-middleware>`.
|
||||
|
||||
They are not necessarily the *first* requests sent, and may not be sent in
|
||||
order; reaching :setting:`CONCURRENT_REQUESTS` and :ref:`scheduling
|
||||
<topics-scheduler>` start requests is prioritized by default. But you can
|
||||
change that.
|
||||
|
||||
..
|
||||
The request send order when all start requests and callback requests have
|
||||
|
|
@ -454,8 +457,8 @@ is reached, start requests are removed from the scheduler immediately after
|
|||
they are scheduled.
|
||||
|
||||
If you do not yield the highest-priority start requests first, and you want
|
||||
request priorities to be respected since the first request, pause the scheduler
|
||||
while yielding your start requests:
|
||||
request priorities to be respected since the very first request, pause the
|
||||
:ref:`scheduler <topics-scheduler>` while yielding your start requests:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue