diff --git a/docs/news.rst b/docs/news.rst index 190e73a9b..9e826f71a 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -107,14 +107,15 @@ New features :meth:`~scrapy.core.engine.ExecutionEngine.needs_backout`. - You can now raise :exc:`~scrapy.exceptions.CloseSpider` from - :meth:`~scrapy.Spider.yield_seeds` and from - :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_seeds`. + :meth:`~scrapy.Spider.start` and from + :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start`. (:issue:`3463`, :issue:`4058`, :issue:`6148`, :issue:`6715`, :issue:`6728`) -- Unlike its precedesors, if :meth:`~scrapy.Spider.yield_seeds` or - :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_seeds` are not - asynchronous generators, the crawl starts nonetheless, without seeds. +- Unlike its precedesors, if :meth:`~scrapy.Spider.start` or + :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start` are not + asynchronous generators, the crawl starts nonetheless, without start + requests. This aligns with the behavior of spider callbacks, and allows the crawl to process requests from the scheduler, e.g. when :ref:`resuming a paused job diff --git a/docs/topics/optimize.rst b/docs/topics/optimize.rst index 3aa2b688b..59d3a1089 100644 --- a/docs/topics/optimize.rst +++ b/docs/topics/optimize.rst @@ -85,11 +85,8 @@ Lowering memory usage :attr:`~scrapy.Request.callback` cannot yield additional requests. - - If you have many seed requests (see - :meth:`~scrapy.Spider.yield_seeds`), set - :setting:`SEEDING_POLICY` to - :class:`~scrapy.SeedingPolicy.lazy` or - :class:`~scrapy.SeedingPolicy.idle`. + - If you have many :ref:`start requests `, use :ref:`lazy + scheduling `. - Set :setting:`JOBDIR` to offload all scheduled requests to disk. diff --git a/docs/topics/scheduler.rst b/docs/topics/scheduler.rst index 38adee1f7..54b72ffc5 100644 --- a/docs/topics/scheduler.rst +++ b/docs/topics/scheduler.rst @@ -26,10 +26,11 @@ A scheduler must: The built-in scheduler stores requests in memory or disk. Other schedulers may rely, for example, on frontier, queue, database or storage services. - Pending requests may come from seeding (see :setting:`SEEDING_POLICY`), - spider callbacks (:attr:`Request.callback `), - :ref:`spider middlewares ` or :ref:`downloader - middlewares `. + Pending requests may come from the + :ref:`Spider.start ` method, from spider + callbacks (:attr:`Request.callback `), + from :ref:`spider middlewares ` or from + :ref:`downloader middlewares `. - Drop unwanted requests. diff --git a/docs/topics/spiders.rst b/docs/topics/spiders.rst index ae80f347d..867331a13 100644 --- a/docs/topics/spiders.rst +++ b/docs/topics/spiders.rst @@ -369,9 +369,14 @@ See `Scrapyd documentation`_. Start requests ============== -Scrapy does not try to send :meth:`~scrapy.Spider.start` requests in order. -Instead, it prioritizes reaching :setting:`CONCURRENT_REQUESTS` and -:ref:`scheduling ` start requests. +**Start requests** are the requests yielded by the :meth:`~scrapy.Spider.start` +method of a spider. + +By default, Scrapy does not try to send :meth:`~scrapy.Spider.start` requests +in order. Instead, it prioritizes reaching :setting:`CONCURRENT_REQUESTS` and +:ref:`scheduling ` start requests. But you can :ref:`force a +specific order ` or :ref:`delay scheduling +` as needed. .. The request send order when all start requests and callback requests have @@ -402,12 +407,12 @@ Instead, it prioritizes reaching :setting:`CONCURRENT_REQUESTS` and .. _start-requests-order: -Forcing a start request order ------------------------------ +Start request order +------------------- -To force a specific **request order**, override the -:meth:`~scrapy.Spider.start` method to set :attr:`Request.priority -`. For example: +To force a specific request order, override the :meth:`~scrapy.Spider.start` +method to set :attr:`Request.priority `. For +example: - To send start requests before other requests: @@ -436,11 +441,12 @@ more control over request prioritization. .. _start-requests-lazy: -Delaying start request iteration --------------------------------- +Lazy start request scheduling +----------------------------- -You can override the :meth:`~scrapy.Spider.start` method as follows to pause -its iteration whenever there are scheduled requests: +To use lazy start request scheduling, where the iteration of start requests is +paused until the scheduler is empty, override the :meth:`~scrapy.Spider.start` +method as follows: .. code-block:: python diff --git a/scrapy/core/scheduler.py b/scrapy/core/scheduler.py index c89d0b2f2..098b7d6e4 100644 --- a/scrapy/core/scheduler.py +++ b/scrapy/core/scheduler.py @@ -87,7 +87,7 @@ class BaseScheduler(metaclass=BaseSchedulerMeta): requests from a slow resource, like a network service, instead of a custom scheduler, consider writing a :ref:`spider middleware ` that implements - :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_seeds`. + :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start`. .. warning:: The crawl will continue running as long as this method returns ``True``. @@ -98,8 +98,8 @@ class BaseScheduler(metaclass=BaseSchedulerMeta): """Called after the spider opens. Useful for initialization code that needs to run later than the - ``__init__`` method, e.g. once seed iteration (see - :meth:`~scrapy.Spider.yield_seeds`) has started. + ``__init__`` method, e.g. once the iteration of the + :meth:`~scrapy.Spider.start` method has started. May return a :class:`~twisted.internet.defer.Deferred`. """ diff --git a/scrapy/exceptions.py b/scrapy/exceptions.py index e3a652728..7f55fd6a5 100644 --- a/scrapy/exceptions.py +++ b/scrapy/exceptions.py @@ -52,9 +52,9 @@ class CloseSpider(Exception): - The :ref:`callbacks ` and the - :meth:`~scrapy.Spider.yield_seeds` method of spiders. + :meth:`~scrapy.Spider.start` method of spiders. - - The :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_seeds` + - The :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start` method of :ref:`spider middlewares `. - :ref:`Handlers ` of the :signal:`spider_idle` signal. diff --git a/tests/test_engine_loop.py b/tests/test_engine_loop.py index e5e969841..b6b1bdcbf 100644 --- a/tests/test_engine_loop.py +++ b/tests/test_engine_loop.py @@ -389,7 +389,7 @@ class RequestSendOrderTestCase(TestCase): def track_url(request, spider): actual_urls.append(request.url) - settings = {"SCHEDULER": TestScheduler, "SEEDING_POLICY": "idle"} + settings = {"SCHEDULER": TestScheduler} crawler = get_crawler(TestSpider, settings_dict=settings) crawler.signals.connect(track_url, signals.request_reached_downloader) await maybe_deferred_to_future(crawler.crawl()) diff --git a/tests/test_spidermiddleware_process_start.py b/tests/test_spidermiddleware_process_start.py index 985a70cae..95bcf48d8 100644 --- a/tests/test_spidermiddleware_process_start.py +++ b/tests/test_spidermiddleware_process_start.py @@ -345,7 +345,7 @@ class MainTestCase(TestCase): @deferred_f_from_coro_f async def test_async_function(self): - async def process_start(mw, seeds): + async def process_start(mw, start): return with LogCapture() as log: @@ -378,7 +378,7 @@ class MainTestCase(TestCase): @deferred_f_from_coro_f async def test_exception_before_yield(self): - async def process_start(mw, seeds): + async def process_start(mw, start): raise RuntimeError yield # pylint: disable=unreachable