RomanNum3ral
/
scrapy
mirror of https://github.com/scrapy/scrapy.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Document the SPIDER_MODULES hack to lower startup time and memory usage (#7600)

This commit is contained in:
Adrian 2026-06-11 12:10:49 +02:00 committed by GitHub
parent e0a7de7213
commit c99f6e2209
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 20 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

20
docs/topics/practices.rst
View File

@ -387,6 +387,26 @@ crawl::
curl http://scrapy2.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=2
curl http://scrapy3.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=3
.. _large-project-startup:
Reducing startup time in large projects
=======================================
When running a spider with ``scrapy crawl``, Scrapy loads all modules listed in
:setting:`SPIDER_MODULES` to find the target spider. In large projects with
many spiders, this can noticeably increase startup time and memory usage.
To avoid loading every spider module, override :setting:`SPIDER_MODULES` on the
command line to point only to the module that contains the spider you want to
run:
.. code-block:: shell
scrapy crawl myspider -s SPIDER_MODULES=myproject.spiders.myspider
Because :setting:`SPIDER_MODULES` is a list setting, you can include multiple
modules by separating them with commas.
.. _bans:
Avoiding getting banned