From c99f6e22094ef1ba65a2015c10badcb685779a55 Mon Sep 17 00:00:00 2001 From: Adrian Date: Thu, 11 Jun 2026 12:10:49 +0200 Subject: [PATCH] Document the SPIDER_MODULES hack to lower startup time and memory usage (#7600) --- docs/topics/practices.rst | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/docs/topics/practices.rst b/docs/topics/practices.rst index 11c2656da..b3c58d6d3 100644 --- a/docs/topics/practices.rst +++ b/docs/topics/practices.rst @@ -387,6 +387,26 @@ crawl:: curl http://scrapy2.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=2 curl http://scrapy3.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=3 +.. _large-project-startup: + +Reducing startup time in large projects +======================================= + +When running a spider with ``scrapy crawl``, Scrapy loads all modules listed in +:setting:`SPIDER_MODULES` to find the target spider. In large projects with +many spiders, this can noticeably increase startup time and memory usage. + +To avoid loading every spider module, override :setting:`SPIDER_MODULES` on the +command line to point only to the module that contains the spider you want to +run: + +.. code-block:: shell + + scrapy crawl myspider -s SPIDER_MODULES=myproject.spiders.myspider + +Because :setting:`SPIDER_MODULES` is a list setting, you can include multiple +modules by separating them with commas. + .. _bans: Avoiding getting banned