From c690eac770a9fb803cdf0003ac3dc73e409146dc Mon Sep 17 00:00:00 2001 From: Adrian Date: Thu, 25 Jun 2026 13:34:37 +0200 Subject: [PATCH] =?UTF-8?q?Document=20=E2=80=9Clogging=20settings=E2=80=9D?= =?UTF-8?q?=20as=20special=20settings=20(#7668)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/topics/practices.rst | 8 ++--- docs/topics/settings.rst | 64 ++++++++++++++++++++++++++++++++++++++- 2 files changed, 67 insertions(+), 5 deletions(-) diff --git a/docs/topics/practices.rst b/docs/topics/practices.rst index 8a04f7ada..e7faf48c4 100644 --- a/docs/topics/practices.rst +++ b/docs/topics/practices.rst @@ -347,10 +347,10 @@ finishes before starting the next one: install_reactor("twisted.internet.asyncioreactor.AsyncioSelectorReactor") react(deferred_f_from_coro_f(crawl)) -.. note:: When running multiple spiders in the same process, :ref:`reactor - settings ` should not have a different value per spider. - Also, :ref:`pre-crawler settings ` cannot be defined - per spider. +.. note:: When running multiple spiders in the same process, :ref:`logging + settings ` and :ref:`reactor settings ` + should not have a different value per spider, and :ref:`pre-crawler + settings ` cannot be defined per spider. .. seealso:: :ref:`run-from-script`. diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst index 455985a56..c452f88d1 100644 --- a/docs/topics/settings.rst +++ b/docs/topics/settings.rst @@ -361,6 +361,31 @@ All of these settings, except for :setting:`ASYNCIO_EVENT_LOOP`, are only used when the Twisted reactor is used, i.e. when :setting:`TWISTED_REACTOR_ENABLED` is ``True``. +.. _logging-settings: + +Logging settings +---------------- + +**Logging settings** are settings that configure the global root logging +handler installed by :func:`~scrapy.utils.log.configure_logging`. + +These settings can be defined from a spider. However, because only 1 root +logging handler is active per process, these settings cannot use a different +value per spider when :ref:`running multiple spiders in the same process +`. + +These settings are: + +- :setting:`LOG_DATEFORMAT` +- :setting:`LOG_ENABLED` +- :setting:`LOG_ENCODING` +- :setting:`LOG_FILE` +- :setting:`LOG_FILE_APPEND` +- :setting:`LOG_FORMAT` +- :setting:`LOG_LEVEL` +- :setting:`LOG_SHORT_NAMES` +- :setting:`LOG_STDOUT` + .. _topics-settings-ref: Built-in settings reference @@ -481,6 +506,8 @@ Note that the event loop class must inherit from :class:`asyncio.AbstractEventLo :func:`asyncio.set_event_loop`, which will set the specified event loop as the current loop for the current OS thread. +.. note:: This is a :ref:`reactor setting `. + .. setting:: BOT_NAME BOT_NAME @@ -662,6 +689,8 @@ Whether to enable DNS in-memory cache. :setting:`TWISTED_REACTOR_ENABLED` is ``False``, and may have no effect either when :setting:`DNS_RESOLVER` is set to a different resolver. +.. note:: This is a :ref:`reactor setting `. + .. setting:: DNSCACHE_SIZE DNSCACHE_SIZE @@ -671,6 +700,8 @@ Default: ``10000`` DNS in-memory cache size, see :setting:`DNSCACHE_ENABLED`. +.. note:: This is a :ref:`reactor setting `. + .. setting:: TWISTED_DNS_RESOLVER TWISTED_DNS_RESOLVER @@ -688,6 +719,8 @@ take the :setting:`DNS_TIMEOUT` setting into account. .. note:: This setting has no effect when :setting:`TWISTED_REACTOR_ENABLED` is ``False``. +.. note:: This is a :ref:`reactor setting `. + .. setting:: DNS_TIMEOUT DNS_TIMEOUT @@ -703,6 +736,8 @@ Timeout for processing of DNS queries in seconds. Float is supported. :setting:`TWISTED_REACTOR_ENABLED` is ``False``, and may have no effect either when :setting:`DNS_RESOLVER` is set to a different resolver. +.. note:: This is a :ref:`reactor setting `. + .. setting:: DOWNLOADER DOWNLOADER @@ -1442,6 +1477,8 @@ Default: ``True`` Whether to enable logging. +.. note:: This is a :ref:`logging setting `. + .. setting:: LOG_ENCODING LOG_ENCODING @@ -1451,6 +1488,8 @@ Default: ``'utf-8'`` The encoding to use for logging. +.. note:: This is a :ref:`logging setting `. + .. setting:: LOG_FILE LOG_FILE @@ -1460,6 +1499,8 @@ Default: ``None`` File name to use for logging output. If ``None``, standard error will be used. +.. note:: This is a :ref:`logging setting `. + .. setting:: LOG_FILE_APPEND LOG_FILE_APPEND @@ -1470,6 +1511,8 @@ Default: ``True`` If ``False``, the log file specified with :setting:`LOG_FILE` will be overwritten (discarding the output from previous runs, if any). +.. note:: This is a :ref:`logging setting `. + .. setting:: LOG_FORMAT LOG_FORMAT @@ -1481,6 +1524,8 @@ String for formatting log messages. Refer to the :ref:`Python logging documentation ` for the whole list of available placeholders. +.. note:: This is a :ref:`logging setting `. + .. setting:: LOG_DATEFORMAT LOG_DATEFORMAT @@ -1493,6 +1538,8 @@ in :setting:`LOG_FORMAT`. Refer to the :ref:`Python datetime documentation ` for the whole list of available directives. +.. note:: This is a :ref:`logging setting `. + .. setting:: LOG_FORMATTER LOG_FORMATTER @@ -1512,6 +1559,8 @@ Default: ``'DEBUG'`` Minimum level to log. Available levels are: CRITICAL, ERROR, WARNING, INFO, DEBUG. For more info see :ref:`topics-logging`. +.. note:: This is a :ref:`logging setting `. + .. setting:: LOG_STDOUT LOG_STDOUT @@ -1523,6 +1572,8 @@ If ``True``, all standard output (and error) of your process will be redirected to the log. For example if you ``print('hello')`` it will appear in the Scrapy log. +.. note:: This is a :ref:`logging setting `. + .. setting:: LOG_SHORT_NAMES LOG_SHORT_NAMES @@ -1533,6 +1584,8 @@ Default: ``False`` If ``True``, the logs will just contain the root path. If it is set to ``False`` then it displays the component responsible for the log output +.. note:: This is a :ref:`logging setting `. + .. setting:: LOG_VERSIONS LOG_VERSIONS @@ -1695,6 +1748,8 @@ multi-purpose thread pool used by various Scrapy components. Threaded DNS Resolver, BlockingFeedStorage, S3FilesStore just to name a few. Increase this value if you're experiencing problems with insufficient blocking IO. +.. note:: This is a :ref:`reactor setting `. + .. setting:: REDIRECT_PRIORITY_ADJUST REDIRECT_PRIORITY_ADJUST @@ -1921,6 +1976,8 @@ Default: ``'scrapy.spiderloader.SpiderLoader'`` The class that will be used for loading spiders, which must implement the :ref:`topics-api-spiderloader`. +.. note:: This is a :ref:`pre-crawler setting `. + .. setting:: SPIDER_LOADER_WARN_ONLY SPIDER_LOADER_WARN_ONLY @@ -1933,6 +1990,8 @@ it will fail loudly if there is any ``ImportError`` or ``SyntaxError`` exception But you can choose to silence this exception and turn it into a simple warning by setting ``SPIDER_LOADER_WARN_ONLY = True``. +.. note:: This is a :ref:`pre-crawler setting `. + .. setting:: SPIDER_MIDDLEWARES SPIDER_MIDDLEWARES @@ -1978,6 +2037,8 @@ Example: SPIDER_MODULES = ["mybot.spiders_prod", "mybot.spiders_dev"] +.. note:: This is a :ref:`pre-crawler setting `. + .. setting:: STATS_CLASS STATS_CLASS @@ -2049,7 +2110,7 @@ stopped) will not apply. This mode is currently experimental and may not be suitable for production use. It may also not be supported by 3rd-party code. See :ref:`asyncio-without-reactor` for more information about this mode. -.. note:: This setting can't be set :ref:`per-spider `. +.. note:: This is a :ref:`pre-crawler setting `. .. versionadded:: 2.15.0 @@ -2156,6 +2217,7 @@ current platform. For additional information, see :doc:`core/howto/choosing-reactor`. +.. note:: This is a :ref:`reactor setting `. .. setting:: URLLENGTH_LIMIT