feat(AI): improved AMD GPU acceleration for Ollama via ROCm + HSA override (#804)
* feat(AI): re-enable AMD GPU acceleration for Ollama via ROCm + HSA override
Re-enables AMD GPU support that was disabled in 77f1868 pending validation
of the ROCm image and device discovery. Validation done 2026-04-28 on a
Minisforum UM890 Pro (Ryzen 9 PRO 8945HS + Radeon 780M iGPU) — Ollama
correctly offloaded all model layers to the iGPU when the container was
started with /dev/kfd + /dev/dri passthrough and HSA_OVERRIDE_GFX_VERSION=11.0.0.
On llama3.2:1b, GPU inference ran at 51.83 tok/s vs 33.16 tok/s on CPU
(same hardware, same prompt) — a 1.56x speedup confirmed by Ollama logs
showing "load_tensors: offloaded 17/17 layers to GPU".
Changes
-------
docker_service.ts
- Restore _discoverAMDDevices() (simplified — pass /dev/dri as a directory
entry, mirroring `docker run --device /dev/dri` behavior, instead of the
prior brittle hardcoded card0/renderD128 fallback that broke on systems
where the AMD GPU enumerates as card1+).
- Restore the AMD branch in _createContainer():
- Switches Ollama image to ollama/ollama:rocm
- Mounts /dev/kfd + /dev/dri via Devices
- Sets HSA_OVERRIDE_GFX_VERSION=11.0.0 (required for unsupported-but-RDNA3
iGPUs like gfx1103; harmless on supported discrete cards)
- KV opt-out via ai.amdGpuAcceleration (default on)
- Mirror the AMD branch in updateContainer():
- Lifted GPU detection above docker.pull() so AMD updates pull :rocm
rather than the standard :targetVersion tag (per-version ROCm tags
aren't always published)
- Replaces stale HSA_OVERRIDE in the inspect-captured env on update,
so containers built before this PR pick up the current value
system_service.ts
- New getOllamaInferenceComputeFromLogs() — parses Ollama startup log line
"msg=\"inference compute\" ... library=CUDA|ROCm ..." which Ollama emits
for both NVIDIA and AMD. Catches silent CPU fallback (e.g. NVML death
after update, or HSA_OVERRIDE failure) that the prior nvidia-smi exec
probe couldn't detect.
- gpuHealth refactored to use log parsing as the primary probe for both
vendors, with nvidia-smi exec retained as the NVIDIA-only secondary
path for hardware enrichment when log parsing has no startup line yet.
- AMD path uses gpu.type KV value (persisted by DockerService._detectGPUType)
+ ai.amdGpuAcceleration opt-out to determine hasRocmRuntime.
types/system.ts
- GpuHealthStatus extended additively: hasRocmRuntime + optional gpuVendor.
types/kv_store.ts
- New ai.amdGpuAcceleration boolean (default-on).
settings/models.tsx, settings/system.tsx
- passthrough_failed banner copy now reads vendor from gpuHealth.gpuVendor
("an AMD GPU" vs "an NVIDIA GPU"). Same Fix button hits the same
force-reinstall endpoint, which now configures AMD correctly.
install_nomad.sh
- AMD detection in verify_gpu_setup() upgraded from a strict-positive
"ROCm not currently available" message to "ROCm acceleration will be
configured automatically." Also tightens the lspci match to display
controller classes (avoids false positives from AMD CPU host bridges,
matching the same fix already in DockerService._detectGPUType).
Auto-remediation
----------------
Issue #755 proposes auto-remediation when gpuHealth.status flips to
passthrough_failed (today the user has to click "Fix: Reinstall AI
Assistant"). When that PR lands, AMD coverage falls out for free since
this PR uses the same passthrough_failed status code via the shared
gpuHealth machinery — #755's guard will need to flip from
hasNvidiaRuntime === true to (hasNvidiaRuntime || hasRocmRuntime).
Closes #124 (AMD GPU support).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(AI): detect AMD GPU presence inside admin container via marker file
The admin container doesn't have lspci installed, and AMD GPUs don't register
a Docker runtime the way NVIDIA does — so DockerService._detectGPUType() and
SystemService.gpuHealth had no way to know an AMD GPU was present.
The previous implementation fell through to lspci, which silently failed inside
the admin container, leaving gpu.type unset and gpuHealth stuck at 'no_gpu'
even on systems with an AMD GPU. (NVIDIA worked because Docker registers the
nvidia runtime, which is reachable via dockerInfo.Runtimes from any container.)
Discovered while testing the AMD acceleration patch on a Minisforum UM890 Pro:
the AMD branch in _createContainer() never fired because _detectGPUType()
returned 'none' even on a host with a working /dev/kfd.
Fix
---
install_nomad.sh writes the host-detected GPU type ('nvidia' | 'amd') to a
marker file in the storage volume the admin container already bind-mounts:
/opt/project-nomad/storage/.nomad-gpu-type → /app/storage/.nomad-gpu-type
DockerService._detectGPUType() reads the marker as a secondary probe (after
the Docker runtime check) — covers AMD detection from inside the container
without requiring lspci or a /dev bind mount.
SystemService falls back to the marker file when KV gpu.type is empty so the
System page reflects AMD presence even before the user installs AI Assistant
for the first time. (Without this, the page would say 'no_gpu' until Ollama
was installed, even on hosts with an AMD GPU detected at install time.)
Verified on NOMAD6 (UM890 Pro, Ubuntu 24.04, 780M iGPU): with the marker file
in place and admin restarted, the patch's AMD branch fires correctly on Force
Reinstall AI Assistant. Resulting nomad_ollama runs ollama/ollama:rocm with
/dev/kfd + /dev/dri passthrough and HSA_OVERRIDE_GFX_VERSION=11.0.0; Ollama
logs show 'library=ROCm compute=gfx1100 ... type=iGPU'. NOMAD's in-product
benchmark on the same hardware climbed from 33.8 tok/s (CPU) to 57.3 tok/s
(GPU) — a 1.69x speedup, with TTFT dropping from 148ms to 66ms.
Migration for existing AMD installs
-----------------------------------
Users on an existing NOMAD install with an AMD GPU have no marker file (the
install script wrote it on a fresh install). Two paths get them on the GPU:
1. Re-run install_nomad.sh — writes the marker, no other side effects
2. Manually: echo amd | sudo tee /opt/project-nomad/storage/.nomad-gpu-type
Either then triggers AMD detection on the next AI Assistant install/reinstall.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(AI): pull ollama/ollama:rocm separately when AMD branch overrides image
The pull-if-missing logic in _createContainer ran against service.container_image
(the DB-pinned tag, e.g. ollama/ollama:0.18.2). The AMD branch then overrode
finalImage to ollama/ollama:rocm — but if that image wasn't already local, the
container creation step failed with "no such image: ollama/ollama:rocm".
Caught while validating on NOMAD2 (Ryzen AI 9 HX 370 + Radeon 890M / RDNA 3.5):
the prior end-to-end test on NOMAD6 had silently passed because the rocm image
was already pulled there from an earlier sidecar test, masking the bug.
Fix: inside the AMD branch, after setting finalImage to ollama/ollama:rocm,
run a parallel _checkImageExists + docker.pull dance for the new tag.
Also confirmed via this validation: the same HSA_OVERRIDE_GFX_VERSION=11.0.0
override works on the 890M (gfx1150 / RDNA 3.5) — Ollama logs report
'library=ROCm compute=gfx1100 description="AMD Radeon 890M Graphics"' and
inference runs at 51.68 tok/s (matching the existing X1 Pro published tile
of 51.7 tok/s on the same hardware class). RDNA 3 (780M, gfx1103) and RDNA
3.5 (890M, gfx1150) both use the same override successfully.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* build(Dockerfile): include pciutils for lspci gpu detection fallback
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Jake Turner <jturner@cosmistack.com>
This commit is contained in:
chriscrosstalk
Jake Turner
parent
5517e826aa
commit
73e2115245
|
|
@ -1,7 +1,14 @@
|
|||
FROM node:22-slim AS base
|
||||
|
||||
# Install bash & curl for entrypoint script compatibility, graphicsmagick for pdf2pic, and vips-dev & build-base for sharp
|
||||
RUN apt-get update && apt-get install -y bash curl graphicsmagick libvips-dev build-essential
|
||||
RUN apt-get update && apt-get install -y \
|
||||
bash \
|
||||
curl \
|
||||
graphicsmagick \
|
||||
libvips-dev \
|
||||
build-essential \
|
||||
pciutils \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# All deps stage
|
||||
FROM base AS deps
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ import { KiwixLibraryService } from './kiwix_library_service.js'
|
|||
import { SERVICE_NAMES } from '../../constants/service_names.js'
|
||||
import { exec } from 'child_process'
|
||||
import { promisify } from 'util'
|
||||
// import { readdir } from 'fs/promises'
|
||||
import { readFile } from 'node:fs/promises'
|
||||
import KVStore from '#models/kv_store'
|
||||
import { BROADCAST_CHANNELS } from '../../constants/broadcast.js'
|
||||
import { KIWIX_LIBRARY_CMD } from '../../constants/kiwix.js'
|
||||
|
|
@ -500,6 +500,7 @@ export class DockerService {
|
|||
// GPU-aware configuration for Ollama
|
||||
let finalImage = service.container_image
|
||||
let gpuHostConfig = containerConfig?.HostConfig || {}
|
||||
let amdGpuConfigured = false
|
||||
|
||||
if (service.service_name === SERVICE_NAMES.OLLAMA) {
|
||||
const gpuResult = await this._detectGPUType()
|
||||
|
|
@ -523,16 +524,51 @@ export class DockerService {
|
|||
],
|
||||
}
|
||||
} else if (gpuResult.type === 'amd') {
|
||||
this._broadcast(
|
||||
service.service_name,
|
||||
'gpu-config',
|
||||
`AMD GPU detected. ROCm GPU acceleration is not yet supported in this version — proceeding with CPU-only configuration. GPU support for AMD will be available in a future update.`
|
||||
)
|
||||
logger.warn('[DockerService] AMD GPU detected but ROCm support is not yet enabled. Using CPU-only configuration.')
|
||||
// TODO: Re-enable AMD GPU support once ROCm image and device discovery are validated.
|
||||
// When re-enabling:
|
||||
// 1. Switch image to 'ollama/ollama:rocm'
|
||||
// 2. Restore _discoverAMDDevices() to map /dev/kfd and /dev/dri/* into the container
|
||||
// AMD acceleration is opt-out via the 'ai.amdGpuAcceleration' KV key (default-on).
|
||||
// Per memory feedback: KV values can be string or boolean — coerce explicitly.
|
||||
const amdEnabledRaw = await KVStore.getValue('ai.amdGpuAcceleration')
|
||||
const amdAccelerationEnabled = String(amdEnabledRaw) !== 'false'
|
||||
|
||||
if (amdAccelerationEnabled) {
|
||||
this._broadcast(
|
||||
service.service_name,
|
||||
'gpu-config',
|
||||
`AMD GPU detected. Using ROCm image with /dev/kfd and /dev/dri passthrough...`
|
||||
)
|
||||
|
||||
finalImage = 'ollama/ollama:rocm'
|
||||
|
||||
// The pull-if-missing earlier in this function used service.container_image
|
||||
// (the DB-pinned tag, e.g. ollama/ollama:0.18.2). The AMD branch overrides
|
||||
// to a different tag — so we need to pull :rocm separately if it's not local.
|
||||
const rocmImageExists = await this._checkImageExists(finalImage)
|
||||
if (!rocmImageExists) {
|
||||
this._broadcast(
|
||||
service.service_name,
|
||||
'pulling',
|
||||
`Pulling Docker image ${finalImage}...`
|
||||
)
|
||||
const rocmPullStream = await this.docker.pull(finalImage)
|
||||
await new Promise((res) => this.docker.modem.followProgress(rocmPullStream, res))
|
||||
}
|
||||
|
||||
const amdDevices = await this._discoverAMDDevices()
|
||||
gpuHostConfig = {
|
||||
...gpuHostConfig,
|
||||
Devices: amdDevices,
|
||||
}
|
||||
amdGpuConfigured = true
|
||||
logger.info(
|
||||
`[DockerService] Configured ROCm image and ${amdDevices.length} AMD device entries for Ollama`
|
||||
)
|
||||
} else {
|
||||
this._broadcast(
|
||||
service.service_name,
|
||||
'gpu-config',
|
||||
`AMD GPU detected but acceleration is disabled via ai.amdGpuAcceleration. Using CPU-only configuration.`
|
||||
)
|
||||
logger.info('[DockerService] AMD GPU acceleration disabled by KV opt-out; using CPU-only configuration.')
|
||||
}
|
||||
} else if (gpuResult.toolkitMissing) {
|
||||
this._broadcast(
|
||||
service.service_name,
|
||||
|
|
@ -555,6 +591,12 @@ export class DockerService {
|
|||
if (flashAttentionEnabled !== false) {
|
||||
ollamaEnv.push('OLLAMA_FLASH_ATTENTION=1')
|
||||
}
|
||||
if (amdGpuConfigured) {
|
||||
// RDNA3 iGPUs (gfx1103: 780M, 880M, 890M, ...) aren't on AMD's official ROCm
|
||||
// allowlist but work when forced to identify as gfx1100 via HSA_OVERRIDE_GFX_VERSION.
|
||||
// Harmless on supported discrete cards (gfx1030 RX 6800, etc.) — they ignore the override.
|
||||
ollamaEnv.push('HSA_OVERRIDE_GFX_VERSION=11.0.0')
|
||||
}
|
||||
}
|
||||
|
||||
this._broadcast(
|
||||
|
|
@ -857,7 +899,10 @@ export class DockerService {
|
|||
/**
|
||||
* Detect GPU type and toolkit availability.
|
||||
* Primary: Check Docker runtimes via docker.info() (works from inside containers).
|
||||
* Fallback: lspci for host-based installs and AMD detection.
|
||||
* Secondary: Read /app/storage/.nomad-gpu-type written by install_nomad.sh — needed
|
||||
* for AMD detection because lspci isn't available inside the admin container and
|
||||
* AMD has no Docker runtime registration to query.
|
||||
* Fallback: lspci for host-based installs.
|
||||
*/
|
||||
private async _detectGPUType(): Promise<{ type: 'nvidia' | 'amd' | 'none'; toolkitMissing?: boolean }> {
|
||||
try {
|
||||
|
|
@ -874,6 +919,24 @@ export class DockerService {
|
|||
logger.warn(`[DockerService] Could not query Docker info for GPU runtimes: ${error.message}`)
|
||||
}
|
||||
|
||||
// Secondary: install_nomad.sh writes the host-detected GPU type to a marker file in
|
||||
// the storage volume so the admin container (which lacks lspci) can read it.
|
||||
try {
|
||||
const marker = (await readFile('/app/storage/.nomad-gpu-type', 'utf8')).trim()
|
||||
if (marker === 'nvidia') {
|
||||
// Hardware present but Docker doesn't have nvidia runtime → toolkit missing
|
||||
logger.warn('[DockerService] NVIDIA GPU recorded in marker file but NVIDIA Container Toolkit is not installed')
|
||||
return { type: 'none', toolkitMissing: true }
|
||||
}
|
||||
if (marker === 'amd') {
|
||||
logger.info('[DockerService] AMD GPU detected via install-time marker file')
|
||||
await this._persistGPUType('amd')
|
||||
return { type: 'amd' }
|
||||
}
|
||||
} catch {
|
||||
// No marker file — fall through to lspci attempt for host-based installs
|
||||
}
|
||||
|
||||
// Fallback: lspci for host-based installs (not available inside Docker)
|
||||
const execAsync = promisify(exec)
|
||||
|
||||
|
|
@ -937,60 +1000,23 @@ export class DockerService {
|
|||
}
|
||||
|
||||
/**
|
||||
* Discover AMD GPU DRI devices dynamically.
|
||||
* Returns an array of device configurations for Docker.
|
||||
* Build the Docker Devices array for AMD GPU passthrough.
|
||||
*
|
||||
* Returns /dev/kfd (Kernel Fusion Driver, required by ROCm) and /dev/dri (the DRM
|
||||
* device tree). Passing /dev/dri as a single directory entry mirrors Docker CLI
|
||||
* --device behavior — the daemon expands it to all child devices (card*, renderD*)
|
||||
* regardless of how the host enumerates them. This avoids the brittle hardcoded
|
||||
* fallback (card0/renderD128) the prior implementation used, which was wrong on
|
||||
* systems where the AMD GPU enumerates as card1+ (e.g. UM890 Pro 780M iGPU).
|
||||
*/
|
||||
// private async _discoverAMDDevices(): Promise<
|
||||
// Array<{ PathOnHost: string; PathInContainer: string; CgroupPermissions: string }>
|
||||
// > {
|
||||
// try {
|
||||
// const devices: Array<{
|
||||
// PathOnHost: string
|
||||
// PathInContainer: string
|
||||
// CgroupPermissions: string
|
||||
// }> = []
|
||||
|
||||
// // Always add /dev/kfd (Kernel Fusion Driver)
|
||||
// devices.push({
|
||||
// PathOnHost: '/dev/kfd',
|
||||
// PathInContainer: '/dev/kfd',
|
||||
// CgroupPermissions: 'rwm',
|
||||
// })
|
||||
|
||||
// // Discover DRI devices in /dev/dri/
|
||||
// try {
|
||||
// const driDevices = await readdir('/dev/dri')
|
||||
// for (const device of driDevices) {
|
||||
// const devicePath = `/dev/dri/${device}`
|
||||
// devices.push({
|
||||
// PathOnHost: devicePath,
|
||||
// PathInContainer: devicePath,
|
||||
// CgroupPermissions: 'rwm',
|
||||
// })
|
||||
// }
|
||||
// logger.info(
|
||||
// `[DockerService] Discovered ${driDevices.length} DRI devices: ${driDevices.join(', ')}`
|
||||
// )
|
||||
// } catch (error) {
|
||||
// logger.warn(`[DockerService] Could not read /dev/dri directory: ${error.message}`)
|
||||
// // Fallback to common device names if directory read fails
|
||||
// const fallbackDevices = ['card0', 'renderD128']
|
||||
// for (const device of fallbackDevices) {
|
||||
// devices.push({
|
||||
// PathOnHost: `/dev/dri/${device}`,
|
||||
// PathInContainer: `/dev/dri/${device}`,
|
||||
// CgroupPermissions: 'rwm',
|
||||
// })
|
||||
// }
|
||||
// logger.info(`[DockerService] Using fallback DRI devices: ${fallbackDevices.join(', ')}`)
|
||||
// }
|
||||
|
||||
// return devices
|
||||
// } catch (error) {
|
||||
// logger.error(`[DockerService] Error discovering AMD devices: ${error.message}`)
|
||||
// return []
|
||||
// }
|
||||
// }
|
||||
private async _discoverAMDDevices(): Promise<
|
||||
Array<{ PathOnHost: string; PathInContainer: string; CgroupPermissions: string }>
|
||||
> {
|
||||
return [
|
||||
{ PathOnHost: '/dev/kfd', PathInContainer: '/dev/kfd', CgroupPermissions: 'rwm' },
|
||||
{ PathOnHost: '/dev/dri', PathInContainer: '/dev/dri', CgroupPermissions: 'rwm' },
|
||||
]
|
||||
}
|
||||
|
||||
/**
|
||||
* Update a service container to a new image version while preserving volumes and data.
|
||||
|
|
@ -1014,12 +1040,60 @@ export class DockerService {
|
|||
|
||||
this.activeInstallations.add(serviceName)
|
||||
|
||||
// Compute new image string
|
||||
// Compute new image string. AMD-on-Ollama overrides this to the rolling :rocm tag
|
||||
// (set during GPU detection below) since per-version ROCm tags aren't always published.
|
||||
const currentImage = service.container_image
|
||||
const imageBase = currentImage.includes(':')
|
||||
? currentImage.substring(0, currentImage.lastIndexOf(':'))
|
||||
: currentImage
|
||||
const newImage = `${imageBase}:${targetVersion}`
|
||||
let newImage = `${imageBase}:${targetVersion}`
|
||||
|
||||
// GPU detection runs before the pull so AMD updates pull ollama/ollama:rocm rather
|
||||
// than the standard tag. Detection result is reused below when building the new
|
||||
// container config (devices, env). Non-Ollama services skip this entirely.
|
||||
let updatedDeviceRequests: any[] | undefined = undefined
|
||||
let updatedAmdDevices: any[] | undefined = undefined
|
||||
let updatedAmdGpuConfigured = false
|
||||
if (serviceName === SERVICE_NAMES.OLLAMA) {
|
||||
const gpuResult = await this._detectGPUType()
|
||||
if (gpuResult.type === 'nvidia') {
|
||||
this._broadcast(
|
||||
serviceName,
|
||||
'update-gpu-config',
|
||||
`NVIDIA container runtime detected. Configuring updated container with GPU support...`
|
||||
)
|
||||
updatedDeviceRequests = [
|
||||
{ Driver: 'nvidia', Count: -1, Capabilities: [['gpu']] },
|
||||
]
|
||||
} else if (gpuResult.type === 'amd') {
|
||||
const amdEnabledRaw = await KVStore.getValue('ai.amdGpuAcceleration')
|
||||
const amdAccelerationEnabled = String(amdEnabledRaw) !== 'false'
|
||||
if (amdAccelerationEnabled) {
|
||||
this._broadcast(
|
||||
serviceName,
|
||||
'update-gpu-config',
|
||||
`AMD GPU detected. Using ROCm image with /dev/kfd and /dev/dri passthrough...`
|
||||
)
|
||||
newImage = 'ollama/ollama:rocm'
|
||||
updatedAmdDevices = await this._discoverAMDDevices()
|
||||
updatedAmdGpuConfigured = true
|
||||
} else {
|
||||
this._broadcast(
|
||||
serviceName,
|
||||
'update-gpu-config',
|
||||
`AMD GPU detected but acceleration is disabled via ai.amdGpuAcceleration. Using CPU-only configuration.`
|
||||
)
|
||||
}
|
||||
} else if (gpuResult.toolkitMissing) {
|
||||
this._broadcast(
|
||||
serviceName,
|
||||
'update-gpu-config',
|
||||
`NVIDIA GPU detected but NVIDIA Container Toolkit is not installed. Using CPU-only configuration. Install the toolkit and reinstall AI Assistant for GPU acceleration: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html`
|
||||
)
|
||||
} else {
|
||||
this._broadcast(serviceName, 'update-gpu-config', `No GPU detected. Using CPU-only configuration.`)
|
||||
}
|
||||
}
|
||||
|
||||
// Step 1: Pull new image
|
||||
this._broadcast(serviceName, 'update-pulling', `Pulling image ${newImage}...`)
|
||||
|
|
@ -1054,48 +1128,21 @@ export class DockerService {
|
|||
|
||||
const hostConfig = inspectData.HostConfig || {}
|
||||
|
||||
// Re-run GPU detection for Ollama so updates always reflect the current GPU environment.
|
||||
// This handles cases where the NVIDIA Container Toolkit was installed after the initial
|
||||
// Ollama setup, and ensures DeviceRequests are always built fresh rather than relying on
|
||||
// round-tripping the Docker inspect format back into the create API.
|
||||
let updatedDeviceRequests: any[] | undefined = undefined
|
||||
if (serviceName === SERVICE_NAMES.OLLAMA) {
|
||||
const gpuResult = await this._detectGPUType()
|
||||
|
||||
if (gpuResult.type === 'nvidia') {
|
||||
this._broadcast(
|
||||
serviceName,
|
||||
'update-gpu-config',
|
||||
`NVIDIA container runtime detected. Configuring updated container with GPU support...`
|
||||
)
|
||||
updatedDeviceRequests = [
|
||||
{
|
||||
Driver: 'nvidia',
|
||||
Count: -1,
|
||||
Capabilities: [['gpu']],
|
||||
},
|
||||
// GPU detection already ran above (before the pull) so we know the right image, devices,
|
||||
// and whether HSA_OVERRIDE needs injection. For AMD, replace any prior HSA_OVERRIDE in
|
||||
// the inspect-captured env so updates from older containers pick up the current value.
|
||||
const baseEnv = inspectData.Config?.Env || []
|
||||
const finalEnv = updatedAmdGpuConfigured
|
||||
? [
|
||||
...baseEnv.filter((e: string) => !e.startsWith('HSA_OVERRIDE_GFX_VERSION=')),
|
||||
'HSA_OVERRIDE_GFX_VERSION=11.0.0',
|
||||
]
|
||||
} else if (gpuResult.type === 'amd') {
|
||||
this._broadcast(
|
||||
serviceName,
|
||||
'update-gpu-config',
|
||||
`AMD GPU detected. ROCm GPU acceleration is not yet supported — using CPU-only configuration.`
|
||||
)
|
||||
} else if (gpuResult.toolkitMissing) {
|
||||
this._broadcast(
|
||||
serviceName,
|
||||
'update-gpu-config',
|
||||
`NVIDIA GPU detected but NVIDIA Container Toolkit is not installed. Using CPU-only configuration. Install the toolkit and reinstall AI Assistant for GPU acceleration: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html`
|
||||
)
|
||||
} else {
|
||||
this._broadcast(serviceName, 'update-gpu-config', `No GPU detected. Using CPU-only configuration.`)
|
||||
}
|
||||
}
|
||||
: baseEnv
|
||||
|
||||
const newContainerConfig: any = {
|
||||
Image: newImage,
|
||||
name: serviceName,
|
||||
Env: inspectData.Config?.Env || undefined,
|
||||
Env: finalEnv.length > 0 ? finalEnv : undefined,
|
||||
Cmd: inspectData.Config?.Cmd || undefined,
|
||||
ExposedPorts: inspectData.Config?.ExposedPorts || undefined,
|
||||
WorkingDir: inspectData.Config?.WorkingDir || undefined,
|
||||
|
|
@ -1105,7 +1152,7 @@ export class DockerService {
|
|||
PortBindings: hostConfig.PortBindings || undefined,
|
||||
RestartPolicy: hostConfig.RestartPolicy || undefined,
|
||||
DeviceRequests: serviceName === SERVICE_NAMES.OLLAMA ? updatedDeviceRequests : (hostConfig.DeviceRequests || undefined),
|
||||
Devices: hostConfig.Devices || undefined,
|
||||
Devices: serviceName === SERVICE_NAMES.OLLAMA && updatedAmdDevices ? updatedAmdDevices : (hostConfig.Devices || undefined),
|
||||
},
|
||||
NetworkingConfig: inspectData.NetworkSettings?.Networks
|
||||
? {
|
||||
|
|
|
|||
|
|
@ -12,6 +12,7 @@ import {
|
|||
} from '../../types/system.js'
|
||||
import { SERVICE_NAMES } from '../../constants/service_names.js'
|
||||
import { readFileSync } from 'node:fs'
|
||||
import { readFile } from 'node:fs/promises'
|
||||
import path, { join } from 'node:path'
|
||||
import { getAllFilesystems, getFile } from '../utils/fs.js'
|
||||
import axios from 'axios'
|
||||
|
|
@ -72,6 +73,61 @@ export class SystemService {
|
|||
return false
|
||||
}
|
||||
|
||||
/**
|
||||
* Probe Ollama startup logs for the canonical "inference compute" line that records
|
||||
* which compute backend was selected. This catches silent CPU fallback (e.g. when
|
||||
* /dev/kfd is mounted but ROCm initialization fails, or NVML dies after an update)
|
||||
* which the older nvidia-smi exec probe could not detect.
|
||||
*
|
||||
* Returns the parsed library, GPU model name, and VRAM in MiB, or null when:
|
||||
* - the Ollama container is not running
|
||||
* - the line has not been emitted (Ollama still starting up)
|
||||
* - logs show CPU-only operation (no GPU detected)
|
||||
*/
|
||||
async getOllamaInferenceComputeFromLogs(): Promise<{
|
||||
library: 'CUDA' | 'ROCm'
|
||||
name: string
|
||||
vramMiB: number
|
||||
} | null> {
|
||||
try {
|
||||
const containers = await this.dockerService.docker.listContainers({ all: false })
|
||||
const ollamaContainer = containers.find((c) => c.Names.includes(`/${SERVICE_NAMES.OLLAMA}`))
|
||||
if (!ollamaContainer) return null
|
||||
|
||||
const container = this.dockerService.docker.getContainer(ollamaContainer.Id)
|
||||
const buf = (await container.logs({
|
||||
stdout: true,
|
||||
stderr: true,
|
||||
tail: 500,
|
||||
follow: false,
|
||||
})) as unknown as Buffer
|
||||
const logs = buf.toString('utf8')
|
||||
|
||||
const lines = logs.split('\n').filter((l) => l.includes('msg="inference compute"'))
|
||||
if (lines.length === 0) return null
|
||||
|
||||
const lastLine = lines[lines.length - 1]
|
||||
const libraryMatch = lastLine.match(/library=(CUDA|ROCm)/)
|
||||
if (!libraryMatch) return null
|
||||
|
||||
const descMatch = lastLine.match(/description="([^"]+)"/)
|
||||
const totalMatch = lastLine.match(/total="([0-9.]+)\s*GiB"/)
|
||||
|
||||
return {
|
||||
library: libraryMatch[1] as 'CUDA' | 'ROCm',
|
||||
name:
|
||||
descMatch?.[1] ||
|
||||
(libraryMatch[1] === 'CUDA' ? 'NVIDIA GPU' : 'AMD GPU'),
|
||||
vramMiB: totalMatch ? Math.round(Number.parseFloat(totalMatch[1]) * 1024) : 0,
|
||||
}
|
||||
} catch (error) {
|
||||
logger.warn(
|
||||
`[SystemService] Failed to probe Ollama logs for inference compute line: ${error instanceof Error ? error.message : error}`
|
||||
)
|
||||
return null
|
||||
}
|
||||
}
|
||||
|
||||
async getNvidiaSmiInfo(): Promise<
|
||||
| Array<{ vendor: string; model: string; vram: number }>
|
||||
| { error: string }
|
||||
|
|
@ -317,10 +373,14 @@ export class SystemService {
|
|||
logger.error('Error reading disk info file:', error)
|
||||
}
|
||||
|
||||
// GPU health tracking — detect when host has NVIDIA GPU but Ollama can't access it
|
||||
// GPU health tracking — detect when host has a GPU runtime but Ollama can't access it.
|
||||
// Primary probe: parse Ollama's "inference compute" startup log line for both NVIDIA
|
||||
// and AMD. Secondary probe (NVIDIA only): nvidia-smi exec, retained as a fallback for
|
||||
// hardware enrichment when log parsing has not yet captured a startup line.
|
||||
let gpuHealth: GpuHealthStatus = {
|
||||
status: 'no_gpu',
|
||||
hasNvidiaRuntime: false,
|
||||
hasRocmRuntime: false,
|
||||
ollamaGpuAccessible: false,
|
||||
}
|
||||
|
||||
|
|
@ -340,27 +400,51 @@ export class SystemService {
|
|||
}
|
||||
|
||||
// If si.graphics() returned no controllers (common inside Docker),
|
||||
// fall back to nvidia runtime + nvidia-smi detection
|
||||
// fall back to runtime + Ollama log probe to figure out what's accessible.
|
||||
if (!graphics.controllers || graphics.controllers.length === 0) {
|
||||
const runtimes = dockerInfo.Runtimes || {}
|
||||
if ('nvidia' in runtimes) {
|
||||
gpuHealth.hasNvidiaRuntime = true
|
||||
const nvidiaInfo = await this.getNvidiaSmiInfo()
|
||||
if (Array.isArray(nvidiaInfo)) {
|
||||
graphics.controllers = nvidiaInfo.map((gpu) => ({
|
||||
model: gpu.model,
|
||||
vendor: gpu.vendor,
|
||||
bus: '',
|
||||
vram: gpu.vram,
|
||||
vramDynamic: false, // assume false here, we don't actually use this field for our purposes.
|
||||
}))
|
||||
gpuHealth.hasNvidiaRuntime = 'nvidia' in runtimes
|
||||
|
||||
// AMD doesn't register a Docker runtime. Detection sources, in priority order:
|
||||
// 1. KV 'gpu.type' (set by DockerService._detectGPUType after first Ollama install)
|
||||
// 2. Marker file at /app/storage/.nomad-gpu-type (written by install_nomad.sh)
|
||||
// The marker file matters because the System page should reflect AMD presence
|
||||
// even before AI Assistant has been installed for the first time.
|
||||
let savedGpuType: string | null | undefined = await KVStore.getValue('gpu.type') as string | undefined
|
||||
if (!savedGpuType) {
|
||||
try {
|
||||
savedGpuType = (await readFile('/app/storage/.nomad-gpu-type', 'utf8')).trim()
|
||||
} catch {}
|
||||
}
|
||||
const amdEnabledRaw = await KVStore.getValue('ai.amdGpuAcceleration')
|
||||
const amdAccelerationEnabled = String(amdEnabledRaw) !== 'false'
|
||||
gpuHealth.hasRocmRuntime = savedGpuType === 'amd' && amdAccelerationEnabled
|
||||
|
||||
if (gpuHealth.hasNvidiaRuntime || gpuHealth.hasRocmRuntime) {
|
||||
gpuHealth.gpuVendor = gpuHealth.hasNvidiaRuntime ? 'nvidia' : 'amd'
|
||||
|
||||
// Primary probe: Ollama log parsing — works for both vendors and catches silent fallback
|
||||
const logInfo = await this.getOllamaInferenceComputeFromLogs()
|
||||
if (logInfo) {
|
||||
graphics.controllers = [
|
||||
{
|
||||
model: logInfo.name,
|
||||
vendor: logInfo.library === 'CUDA' ? 'NVIDIA' : 'AMD',
|
||||
bus: '',
|
||||
vram: logInfo.vramMiB,
|
||||
vramDynamic: false,
|
||||
},
|
||||
]
|
||||
gpuHealth.status = 'ok'
|
||||
gpuHealth.ollamaGpuAccessible = true
|
||||
} else if (nvidiaInfo === 'OLLAMA_NOT_FOUND') {
|
||||
// No local Ollama container — check if a remote Ollama URL is configured
|
||||
const externalOllamaGpu = await this.getExternalOllamaGpuInfo()
|
||||
if (externalOllamaGpu) {
|
||||
graphics.controllers = externalOllamaGpu.map((gpu) => ({
|
||||
} else if (gpuHealth.hasNvidiaRuntime) {
|
||||
// NVIDIA secondary path: nvidia-smi exec preserves prior behavior when
|
||||
// the log parser hasn't seen a startup line yet (e.g. log rotation,
|
||||
// very fresh container). Distinguishes "no Ollama container" from
|
||||
// "container exists but GPU broken".
|
||||
const nvidiaInfo = await this.getNvidiaSmiInfo()
|
||||
if (Array.isArray(nvidiaInfo)) {
|
||||
graphics.controllers = nvidiaInfo.map((gpu) => ({
|
||||
model: gpu.model,
|
||||
vendor: gpu.vendor,
|
||||
bus: '',
|
||||
|
|
@ -369,25 +453,66 @@ export class SystemService {
|
|||
}))
|
||||
gpuHealth.status = 'ok'
|
||||
gpuHealth.ollamaGpuAccessible = true
|
||||
} else if (nvidiaInfo === 'OLLAMA_NOT_FOUND') {
|
||||
const externalOllamaGpu = await this.getExternalOllamaGpuInfo()
|
||||
if (externalOllamaGpu) {
|
||||
graphics.controllers = externalOllamaGpu.map((gpu) => ({
|
||||
model: gpu.model,
|
||||
vendor: gpu.vendor,
|
||||
bus: '',
|
||||
vram: gpu.vram,
|
||||
vramDynamic: false,
|
||||
}))
|
||||
gpuHealth.status = 'ok'
|
||||
gpuHealth.ollamaGpuAccessible = true
|
||||
} else {
|
||||
gpuHealth.status = 'ollama_not_installed'
|
||||
}
|
||||
} else {
|
||||
gpuHealth.status = 'ollama_not_installed'
|
||||
const externalOllamaGpu = await this.getExternalOllamaGpuInfo()
|
||||
if (externalOllamaGpu) {
|
||||
graphics.controllers = externalOllamaGpu.map((gpu) => ({
|
||||
model: gpu.model,
|
||||
vendor: gpu.vendor,
|
||||
bus: '',
|
||||
vram: gpu.vram,
|
||||
vramDynamic: false,
|
||||
}))
|
||||
gpuHealth.status = 'ok'
|
||||
gpuHealth.ollamaGpuAccessible = true
|
||||
} else {
|
||||
gpuHealth.status = 'passthrough_failed'
|
||||
logger.warn(
|
||||
`NVIDIA runtime detected but GPU passthrough failed: ${typeof nvidiaInfo === 'string' ? nvidiaInfo : JSON.stringify(nvidiaInfo)}`
|
||||
)
|
||||
}
|
||||
}
|
||||
} else {
|
||||
const externalOllamaGpu = await this.getExternalOllamaGpuInfo()
|
||||
if (externalOllamaGpu) {
|
||||
graphics.controllers = externalOllamaGpu.map((gpu) => ({
|
||||
model: gpu.model,
|
||||
vendor: gpu.vendor,
|
||||
bus: '',
|
||||
vram: gpu.vram,
|
||||
vramDynamic: false,
|
||||
}))
|
||||
gpuHealth.status = 'ok'
|
||||
gpuHealth.ollamaGpuAccessible = true
|
||||
// AMD path: no nvidia-smi equivalent worth running — log parser is authoritative.
|
||||
// Distinguish "Ollama not running" from "Ollama running but no GPU log line".
|
||||
const containers = await this.dockerService.docker.listContainers({ all: false })
|
||||
const ollamaRunning = containers.some((c) =>
|
||||
c.Names.includes(`/${SERVICE_NAMES.OLLAMA}`)
|
||||
)
|
||||
if (!ollamaRunning) {
|
||||
const externalOllamaGpu = await this.getExternalOllamaGpuInfo()
|
||||
if (externalOllamaGpu) {
|
||||
graphics.controllers = externalOllamaGpu.map((gpu) => ({
|
||||
model: gpu.model,
|
||||
vendor: gpu.vendor,
|
||||
bus: '',
|
||||
vram: gpu.vram,
|
||||
vramDynamic: false,
|
||||
}))
|
||||
gpuHealth.status = 'ok'
|
||||
gpuHealth.ollamaGpuAccessible = true
|
||||
} else {
|
||||
gpuHealth.status = 'ollama_not_installed'
|
||||
}
|
||||
} else {
|
||||
gpuHealth.status = 'passthrough_failed'
|
||||
logger.warn(
|
||||
`NVIDIA runtime detected but GPU passthrough failed: ${typeof nvidiaInfo === 'string' ? nvidiaInfo : JSON.stringify(nvidiaInfo)}`
|
||||
'AMD GPU detected but Ollama logs show no ROCm initialization — passthrough or HSA override may have failed'
|
||||
)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -283,7 +283,7 @@ export default function ModelsPage(props: {
|
|||
type="warning"
|
||||
variant="bordered"
|
||||
title="GPU Not Accessible"
|
||||
message={`Your system has an NVIDIA GPU, but ${aiAssistantName} can't access it. AI is running on CPU only, which is significantly slower.`}
|
||||
message={`Your system has ${systemInfo?.gpuHealth?.gpuVendor === 'amd' ? 'an AMD' : 'an NVIDIA'} GPU, but ${aiAssistantName} can't access it. AI is running on CPU only, which is significantly slower.`}
|
||||
className="!mt-6"
|
||||
dismissible={true}
|
||||
onDismiss={handleDismissGpuBanner}
|
||||
|
|
|
|||
|
|
@ -209,7 +209,7 @@ export default function SettingsPage(props: {
|
|||
type="warning"
|
||||
variant="bordered"
|
||||
title="GPU Not Accessible to AI Assistant"
|
||||
message="Your system has an NVIDIA GPU, but the AI Assistant can't access it. AI is running on CPU only, which is significantly slower."
|
||||
message={`Your system has ${info?.gpuHealth?.gpuVendor === 'amd' ? 'an AMD' : 'an NVIDIA'} GPU, but the AI Assistant can't access it. AI is running on CPU only, which is significantly slower.`}
|
||||
dismissible={true}
|
||||
onDismiss={handleDismissGpuBanner}
|
||||
buttonProps={{
|
||||
|
|
|
|||
|
|
@ -12,6 +12,7 @@ export const KV_STORE_SCHEMA = {
|
|||
'gpu.type': 'string',
|
||||
'ai.remoteOllamaUrl': 'string',
|
||||
'ai.ollamaFlashAttention': 'boolean',
|
||||
'ai.amdGpuAcceleration': 'boolean',
|
||||
} as const
|
||||
|
||||
type KVTagToType<T extends string> = T extends 'boolean' ? boolean : string
|
||||
|
|
|
|||
|
|
@ -3,7 +3,9 @@ import { Systeminformation } from 'systeminformation'
|
|||
export type GpuHealthStatus = {
|
||||
status: 'ok' | 'passthrough_failed' | 'no_gpu' | 'ollama_not_installed'
|
||||
hasNvidiaRuntime: boolean
|
||||
hasRocmRuntime: boolean
|
||||
ollamaGpuAccessible: boolean
|
||||
gpuVendor?: 'nvidia' | 'amd'
|
||||
}
|
||||
|
||||
export type SystemInformationResponse = {
|
||||
|
|
|
|||
|
|
@ -517,18 +517,35 @@ verify_gpu_setup() {
|
|||
echo -e "${YELLOW}○${RESET} Docker NVIDIA runtime not detected\\n"
|
||||
fi
|
||||
|
||||
# Check for AMD GPU
|
||||
# Check for AMD GPU — restrict to display controller classes to avoid false positives
|
||||
# from AMD CPU host bridges, PCI bridges, and chipset devices.
|
||||
local has_amd_gpu='false'
|
||||
if command -v lspci &> /dev/null; then
|
||||
if lspci 2>/dev/null | grep -iE "amd|radeon" &> /dev/null; then
|
||||
echo -e "${YELLOW}○${RESET} AMD GPU detected (ROCm support not currently available)\\n"
|
||||
if lspci 2>/dev/null | grep -iE "VGA|3D controller|Display" | grep -iE "amd|radeon" &> /dev/null; then
|
||||
has_amd_gpu='true'
|
||||
echo -e "${GREEN}✓${RESET} AMD GPU detected — ROCm acceleration will be configured automatically when AI Assistant is installed.\\n"
|
||||
fi
|
||||
fi
|
||||
|
||||
|
||||
# Write detected GPU type to a marker file the admin container can read. The admin
|
||||
# container lacks lspci and AMD GPUs don't register a Docker runtime, so this is the
|
||||
# only reliable way for the admin to know an AMD GPU is present at install time.
|
||||
local gpu_marker_path="${NOMAD_DIR}/storage/.nomad-gpu-type"
|
||||
if command -v nvidia-smi &> /dev/null; then
|
||||
echo 'nvidia' | sudo tee "${gpu_marker_path}" > /dev/null 2>&1 || true
|
||||
elif [[ "${has_amd_gpu}" == 'true' ]]; then
|
||||
echo 'amd' | sudo tee "${gpu_marker_path}" > /dev/null 2>&1 || true
|
||||
else
|
||||
sudo rm -f "${gpu_marker_path}" 2>/dev/null || true
|
||||
fi
|
||||
|
||||
echo -e "${YELLOW}===========================================${RESET}\\n"
|
||||
|
||||
|
||||
# Summary
|
||||
if command -v nvidia-smi &> /dev/null && docker info 2>/dev/null | grep -q "nvidia"; then
|
||||
echo -e "${GREEN}#${RESET} GPU acceleration is properly configured! The AI Assistant will use your GPU.\\n"
|
||||
elif [[ "${has_amd_gpu}" == 'true' ]]; then
|
||||
echo -e "${GREEN}#${RESET} GPU acceleration will be enabled (AMD/ROCm) when AI Assistant is installed from the dashboard.\\n"
|
||||
else
|
||||
echo -e "${YELLOW}#${RESET} GPU acceleration not detected. The AI Assistant will run in CPU-only mode.\\n"
|
||||
if command -v nvidia-smi &> /dev/null && ! docker info 2>/dev/null | grep -q "nvidia"; then
|
||||
|
|
|
|||
Loading…
Reference in New Issue