Review uses of seed

docs/news.rst

@@ -107,14 +107,15 @@ New features
     :meth:`~scrapy.core.engine.ExecutionEngine.needs_backout`.
 
 -   You can now raise :exc:`~scrapy.exceptions.CloseSpider` from
-    :meth:`~scrapy.Spider.yield_seeds` and from
-    :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_seeds`.
+    :meth:`~scrapy.Spider.start` and from
+    :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start`.
 
     (:issue:`3463`, :issue:`4058`, :issue:`6148`, :issue:`6715`, :issue:`6728`)
 
--   Unlike its precedesors, if :meth:`~scrapy.Spider.yield_seeds` or
-    :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_seeds` are not
-    asynchronous generators, the crawl starts nonetheless, without seeds.
+-   Unlike its precedesors, if :meth:`~scrapy.Spider.start` or
+    :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start` are not
+    asynchronous generators, the crawl starts nonetheless, without start
+    requests.
 
     This aligns with the behavior of spider callbacks, and allows the crawl to
     process requests from the scheduler, e.g. when :ref:`resuming a paused job

docs/topics/optimize.rst

@@ -85,11 +85,8 @@ Lowering memory usage
         :attr:`~scrapy.Request.callback` cannot yield additional
         requests.
 
-    -   If you have many seed requests (see
-        :meth:`~scrapy.Spider.yield_seeds`), set
-        :setting:`SEEDING_POLICY` to
-        :class:`~scrapy.SeedingPolicy.lazy` or
-        :class:`~scrapy.SeedingPolicy.idle`.
+    -   If you have many :ref:`start requests <start-requests>`, use :ref:`lazy
+        scheduling <start-requests-lazy>`.
 
     -   Set :setting:`JOBDIR` to offload all scheduled requests to disk.
 

docs/topics/scheduler.rst

@@ -26,10 +26,11 @@ A scheduler must:
     The built-in scheduler stores requests in memory or disk. Other schedulers
     may rely, for example, on frontier, queue, database or storage services.
 
-    Pending requests may come from seeding (see :setting:`SEEDING_POLICY`),
-    spider callbacks (:attr:`Request.callback <scrapy.Request.callback>`),
-    :ref:`spider middlewares <topics-spider-middleware>` or :ref:`downloader
-    middlewares <topics-downloader-middleware>`.
+    Pending requests may come from the
+    :ref:`Spider.start <scrapy.spiders.Spider.start>` method, from spider
+    callbacks (:attr:`Request.callback <scrapy.Request.callback>`),
+    from :ref:`spider middlewares <topics-spider-middleware>` or from
+    :ref:`downloader middlewares <topics-downloader-middleware>`.
 
 -   Drop unwanted requests.
 

docs/topics/spiders.rst

@@ -369,9 +369,14 @@ See `Scrapyd documentation`_.
 Start requests
 ==============
 
-Scrapy does not try to send :meth:`~scrapy.Spider.start` requests in order.
-Instead, it prioritizes reaching :setting:`CONCURRENT_REQUESTS` and
-:ref:`scheduling <topics-scheduler>` start requests.
+**Start requests** are the requests yielded by the :meth:`~scrapy.Spider.start`
+method of a spider.
+
+By default, Scrapy does not try to send :meth:`~scrapy.Spider.start` requests
+in order. Instead, it prioritizes reaching :setting:`CONCURRENT_REQUESTS` and
+:ref:`scheduling <topics-scheduler>` start requests. But you can :ref:`force a
+specific order <start-requests-order>` or :ref:`delay scheduling
+<start-requests-lazy>` as needed.
 
 ..
     The request send order when all start requests and callback requests have
@@ -402,12 +407,12 @@ Instead, it prioritizes reaching :setting:`CONCURRENT_REQUESTS` and
 
 .. _start-requests-order:
 
-Forcing a start request order
------------------------------
+Start request order
+-------------------
 
-To force a specific **request order**, override the
-:meth:`~scrapy.Spider.start` method to set :attr:`Request.priority
-<scrapy.http.Request.priority>`. For example:
+To force a specific request order, override the :meth:`~scrapy.Spider.start`
+method to set :attr:`Request.priority <scrapy.http.Request.priority>`. For
+example:
 
 -   To send start requests before other requests:
 
@@ -436,11 +441,12 @@ more control over request prioritization.
 
 .. _start-requests-lazy:
 
-Delaying start request iteration
---------------------------------
+Lazy start request scheduling
+-----------------------------
 
-You can override the :meth:`~scrapy.Spider.start` method as follows to pause
-its iteration whenever there are scheduled requests:
+To use lazy start request scheduling, where the iteration of start requests is
+paused until the scheduler is empty, override the :meth:`~scrapy.Spider.start`
+method as follows:
 
 .. code-block:: python
 

scrapy/core/scheduler.py

@@ -87,7 +87,7 @@ class BaseScheduler(metaclass=BaseSchedulerMeta):
             requests from a slow resource, like a network service, instead of a
             custom scheduler, consider writing a :ref:`spider middleware
             <topics-spider-middleware>` that implements
-            :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_seeds`.
+            :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start`.
 
         .. warning:: The crawl will continue running as long as this method
             returns ``True``.
@@ -98,8 +98,8 @@ class BaseScheduler(metaclass=BaseSchedulerMeta):
         """Called after the spider opens.
 
         Useful for initialization code that needs to run later than the
-        ``__init__`` method, e.g. once seed iteration (see
-        :meth:`~scrapy.Spider.yield_seeds`) has started.
+        ``__init__`` method, e.g. once the iteration of the
+        :meth:`~scrapy.Spider.start` method has started.
 
         May return a :class:`~twisted.internet.defer.Deferred`.
         """

scrapy/exceptions.py

@@ -52,9 +52,9 @@ class CloseSpider(Exception):
 
     -   The :ref:`callbacks
         <topics-request-response-ref-request-callback-arguments>` and the
-        :meth:`~scrapy.Spider.yield_seeds` method of spiders.
+        :meth:`~scrapy.Spider.start` method of spiders.
 
-    -   The :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_seeds`
+    -   The :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start`
         method of :ref:`spider middlewares <topics-spider-middleware>`.
 
     -   :ref:`Handlers <topics-signals>` of the :signal:`spider_idle` signal.

tests/test_engine_loop.py

@@ -389,7 +389,7 @@ class RequestSendOrderTestCase(TestCase):
         def track_url(request, spider):
             actual_urls.append(request.url)
 
-        settings = {"SCHEDULER": TestScheduler, "SEEDING_POLICY": "idle"}
+        settings = {"SCHEDULER": TestScheduler}
         crawler = get_crawler(TestSpider, settings_dict=settings)
         crawler.signals.connect(track_url, signals.request_reached_downloader)
         await maybe_deferred_to_future(crawler.crawl())

tests/test_spidermiddleware_process_start.py

@@ -345,7 +345,7 @@ class MainTestCase(TestCase):
 
     @deferred_f_from_coro_f
     async def test_async_function(self):
-        async def process_start(mw, seeds):
+        async def process_start(mw, start):
             return
 
         with LogCapture() as log:
@@ -378,7 +378,7 @@ class MainTestCase(TestCase):
 
     @deferred_f_from_coro_f
     async def test_exception_before_yield(self):
-        async def process_start(mw, seeds):
+        async def process_start(mw, start):
             raise RuntimeError
             yield  # pylint: disable=unreachable