BullMQ instantiates a fresh ioredis client per Queue/Worker when handed a
plain {host, port} config object, and under sustained ZIM ingestion the
embed pipeline leaked ~1 client/sec until Redis maxclients was exhausted.
Pass a single shared ioredis instance (maxRetriesPerRequest: null, as
required by BullMQ) so all queues and workers reuse one client pool.
Workers still duplicate the connection once for their blocking client,
which is expected and bounded.
Closes#885
* feat: replace legacy Kolibri image default with latest v19 image
* feat(supply-depot): add content migration instructions for Edu Platform Gen 1 to 2
Adds the MeshCore web client to the Supply Depot catalog (host port 8500),
alongside the existing Meshtastic apps. Uses aXistem's prebuilt image of Liam
Cottle's MeshCore client (MeshCore is a sibling LoRa mesh project to Meshtastic).
The image is stock nginx serving a static Flutter build over HTTP, but the
client reaches radios via Web Bluetooth / Web Serial, which browsers only allow
from a secure (HTTPS) context. So we serve it over HTTPS: a new preinstall hook
generates a self-signed cert + a small SSL nginx config into storage/meshcore-web,
both bind-mounted into the container (the config over the image's default.conf),
publishing 443. Same one-time browser-warning approach as Vaultwarden, whose
openssl cert generation is refactored into a shared _ensureSelfSignedCert helper.
Also adds a NOMAD-specific docs section + Manage>Docs anchor, and registers the
IconAntenna icon. Meshtastic Web left unchanged.
Validated on NOMAD3 (v1.33.0-rc.1): the image + SSL config + self-signed cert
serves the MeshCore Flutter app over HTTPS 200 with working SPA fallback.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Curated catalog apps could be installed, stopped, and force-reinstalled,
but never removed — the only path off a device was manual docker + DB
surgery. Custom apps already had delete; this adds the equivalent for
curated apps.
POST /api/system/services/uninstall stops and removes the app's
container (optionally its image, same best-effort semantics as custom
app delete) and flips the record back to not-installed so the card
returns to the available catalog. Host bind-mount data is deliberately
left on disk, so a later reinstall picks the app back up where it left
off — unlike force-reinstall, which clears volumes.
Guards: custom apps are rejected (use delete), dependency services are
rejected, and uninstalling a not-installed app is a 409.
UI: installed curated cards get an Uninstall action in the card menu,
with a confirm modal that explains data is preserved and offers the
same remove-image checkbox as custom app delete.
Refresh the in-app Markdoc docs for the v1.33 feature set:
- Repoint dead /settings/apps links to the Supply Depot (/supply-depot)
across home, getting-started, and faq; reword "Apps page" / "Settings
-> Apps" to "Supply Depot". The old /apps route now redirects to the
Supply Depot.
- Expand supply-depot-apps.md with a "Managing your apps" section (Docs/
Edit/Logs/Stats/Update/Remove, version + update-available visibility,
custom launch URLs, per-app auto-update toggle) and a "Bringing your
own app" section for custom Docker containers.
- Add a new "Updates" doc (updates.md) covering the auto-update trilogy
(core/apps/content), manual updates, and the Early Access channel;
wire it into DOC_ORDER and cross-link from home, getting-started, faq.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Only Wikipedia had version cleanup; every other curated map and non-Wikipedia
ZIM left its prior version on disk when a newer one installed, so users silently
accumulated orphaned content (potentially hundreds of GB). (#634)
The install paths already record each resource via InstalledResource
{resource_id, resource_type, version, file_path}, so the authoritative old-file
path for a resource is known. On install of a new version we now capture the
prior row before updateOrCreate repoints it, then delete the old file — gated
behind a pure, fully unit-tested decision function with strict safety rails:
- tracked-only: requires a prior InstalledResource row for the same
resource_id, so sideloaded/untracked files are never touched
- genuine replacement: old and new file paths must differ
- new-file-verified: the new file must be confirmed on disk first
- strictly-newer: a re-install or downgrade can't wipe a newer file
- within-storage-dir: the old path must resolve under the content store
ZIM cleanup deletes the old file directly (NOT via this.delete(), which would
drop the InstalledResource row by resource_id that updateOrCreate just
repointed) and rebuilds the Kiwix library only if a file was actually removed,
so its XML never references a deleted ZIM. Maps need no library step. Wikipedia
keeps its own existing cleanup path. All deletions are best-effort and logged;
a failure never breaks the install.
Closes#634
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`getChatSuggestions` previously picked the largest installed model by file
size, on the assumption that bigger models give better suggestions. This
is unsafe: if any installed model exceeds available VRAM (e.g.
llama3.1:405b on a 96 GB GPU), Ollama spends minutes trying to load it
and the request 500s — making the chat page unusable for anyone who
happens to keep a flagship-sized model on disk.
Chat suggestions are short prompts that don't benefit from a flagship
model anyway. Prefer the user's selected `chat.lastModel` when set, and
fall back to the smallest installed model otherwise. `OllamaService.getModels()`
already excludes embedders, so the fallback always picks a chat model.
listTags() follows the registry's Link-header pagination, but the next-page
URL is relative per the OCI/Docker registry spec (e.g.
"/v2/ollama/ollama/tags/list?last=0.9.3-rc5&n=1000"). The code assigned that
raw relative path straight back to `url` and re-fetched it, so fetch() threw
"Failed to parse URL from /v2/...". Any image repo with more than 1000 tags
paginates, so the entire tag list — and therefore the update check — failed
silently for ollama/ollama and filebrowser/filebrowser.
That's the root cause of #945 ("won't update past 0.24.0"): the Ollama
update check never completed, so no newer version was ever offered.
Resolve the next-page URL against the registry origin with
new URL(next, `https://${registry}`), which also passes absolute next-URLs
through unchanged for registries that return those.
Closes#945
The map style names each source by its date-stripped region (both
"washington.pmtiles" and "washington_2025-12.pmtiles" -> "washington").
When an old and new copy of the same region are both on disk, the style
emitted two sources with the same key and duplicate layer ids, which
MapLibre rejects outright -- blanking the ENTIRE map, not just that region.
Old copies linger when a newer curated version installs (#634), so a user
who updates maps can silently lose all map rendering until the stale file
is removed by hand.
generateSourcesArray() now keeps only the newest file per region: a dated
build beats an undated legacy file, and between two dated builds the later
YYYY-MM wins. The skipped duplicate is logged. The style stays valid even
when stale files are present.
Complements #981, which removes superseded curated files on install. This
is the runtime safety net that also recovers installs already in the broken
state (which a cleanup-on-install alone can't reach).
Refs #634
Dense source content produces chunks that exceed the embedding model's
context window (nomic-embed-text:v1.5 defaults to 2048 tokens). Two paths
hit this even after the prior pre-cap:
- Older Ollama (e.g. 0.18.1, #944) ignores the num_ctx=8192 we send on
/api/embed, so it stays at the model's 2048 default.
- The OpenAI-compat /v1/embeddings fallback didn't pass num_ctx/truncate
at all, so any Ollama drops to 2048 whenever it lands on the fallback.
When a chunk overflowed, the 400 was swallowed and the chunk was silently
dropped from Qdrant. Worse, the failure propagated to EmbedFileJob, which
re-embeds the entire file on each of its 30 BullMQ attempts — the "endless
queue loop" / "api/embed for weeks" / pegged GPU reported in #944/#959.
Fix:
- OllamaService.embed(): on a context-length error, retry once with an
aggressive 2048-safe cap (EMBED_CONTEXT_SAFE_CHARS = 2000) so the chunk
is embedded (start-of-chunk) instead of dropped. Native-path context
errors now bubble to this retry instead of falling through to the
smaller-context fallback. Split the native+fallback attempt into
_embedWithFallback().
- Pass truncate/num_ctx on the /v1/embeddings fallback too (Ollama's
OpenAI-compat shim forwards them).
- EmbedFileJob: classify "input length exceeds context length" as an
UnrecoverableError so one permanently-oversized chunk can't trigger 30
full-file re-embeds.
- Add OllamaService.isContextLengthError() shared by both.
Graceful degradation: a truncated chunk loses its tail but is kept in the
index, which is strictly better than today's silent drop + retry storm.
Refs #881. Supersedes the #369/#670 symptom closures that never fixed the
fallback path.
reconcileFromFilesystem() skipped every ZIM whose filename starts with
`wikipedia_en_`, on the assumption that all such files are managed by the
WikipediaSelection model. But curated category tiers ship Wikipedia-themed
ZIMs (e.g. Medicine → Comprehensive includes `wikipedia_en_medicine_maxi`),
so those files were skipped during reconcile and their InstalledResource
rows got wiped on every restart — silently downgrading the detected tier.
Skip only the single file actually tracked by WikipediaSelection, matched
by exact filename instead of the `wikipedia_en_` prefix.
Reimplemented in-house from @johno10661's PR #774 (which was trapped on a
stale base); credit to them for the diagnosis and fix.
Closes#774
A multi-GB service update (e.g. nomad_ollama pulling ~6.5 GB) left the
Update button clickable with no feedback, so users clicked again thinking
it was stuck. The second click raced a concurrent updateContainer run into
Docker 304/400 errors (stop/rename on a container the first run had already
moved). The backend lock was in-memory only and never written to the DB, so
nothing durable signaled "update in progress" to the UI, and a page reload
mid-pull re-enabled the button.
Backend (docker_service.updateContainer):
- Set installation_status='installing' when the update starts and reset it
to 'idle' in a finally on every exit path. This mirrors the install path,
survives a page reload, and is visible to other tabs/clients.
- Reject a second update with a clear message when installation_status is
already 'installing', instead of letting it race into Docker errors.
Frontend (settings/apps.tsx):
- Track in-flight updates per service. Seed optimistically on click and
reconcile with the durable installation_status from the server.
- Disable the per-service Update button and show "Updating..." while in
flight. Drop the fullscreen spinner for updates so the table and the
activity feed (live pull/stop/start progress) stay visible.
Closes#931
Kiwix runs in library mode reading kiwix-library.xml via --monitorLibrary.
Today a missing or corrupt XML is only repaired on the download path
(rebuildFromDisk after a completed download), so if the file is lost or
truncated outside that flow — storage relocation, an interrupted write, manual
deletion — Kiwix comes up serving an empty library with no path to recovery.
Add KiwixLibraryService.ensureLibraryXmlHealthy(): reads the XML, and if it's
missing (ENOENT) or fails to parse / lacks a <library> root, rebuilds it from
the ZIM files on disk. A well-formed but empty library is treated as valid (no
spurious rebuild), and filesystem errors other than ENOENT are surfaced rather
than masked. The boot provider calls it on the already-in-library-mode path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Let users override an app's "Open" link with a reverse-proxy or
local-DNS address (e.g. https://jellyfin.myhomelab.net). Falls back to
the default host+port when unset. Metadata-only — no container changes.
Replaces the regex blocklist in assertNotPrivateUrl with ipaddr.js range
classification and normalizes the host before checking it. Consolidates two
community proposals (#930 ipaddr.js parsing, #912 trailing-dot normalization)
into one validator so the SSRF-critical path lives in-house with full tests.
- Classify literal IPs by range (loopback / linkLocal / unspecified) via
ipaddr.js instead of a hand-maintained regex list, which also catches
alternate IPv4 encodings and avoids over-blocking mapped public IPs (the old
`::ffff:` regex blocked every mapped address, including public ones). IPv4-
mapped IPv6 is reduced to its embedded IPv4 before classification.
- Strip a trailing root dot from the host so `localhost.` / `127.0.0.1.` can't
bypass the checks (they resolve to the same target as the dotless form, #911).
- Strip IPv6 brackets and lowercase for the localhost comparison.
- RFC1918, bare LAN hostnames (e.g. `nomad3`), and external FQDNs remain
allowed — LAN appliances need them, and DNS rebinding is a fetch-time concern
outside this guard's scope.
Adds a consolidated unit spec covering loopback/link-local/unspecified literals,
alternate encodings, IPv4-mapped v6, mixed-case + trailing-dot localhost, and
the allowed LAN/FQDN/mapped-public cases.
Resolves#922. Supersedes #930 and #912 (thanks @Gujiassh and @luyua9).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every Docker pull went through `followProgress(pullStream, resolve)`, passing
the Promise's resolve as dockerode's onFinished(err, output) callback — so the
error argument was ignored and a failed pull (dropped/metered connection, bad
manifest, registry error, disk full mid-pull) resolved as if it had succeeded.
The code then tried to create/start a container from a missing or partial
image, surfacing a confusing downstream error rather than the real cause. (#790)
Add a DockerService.pullImage() helper that rejects when followProgress reports
an error, and route all five pull sites through it:
- service install
- AMD ROCm image pull
- service update
- force-reinstall / recreate (forcePull)
- sysbench benchmark image pull
Closes#790
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On ZIMs that pack one logical article as several sub-pages (e.g. iFixit),
iterByPath yields more entries passing our isArticleEntry() filter than
archive.articleCount reports. The inter-batch progress used nextOffset /
articleCount, so the numerator outran the denominator, the ratio overflowed past
100%, and the UI (which clamps at 99%) pinned the file at 99% for the entire
remaining tail, making it look hung.
Grow the denominator to max(articleCount, nextOffset + ZIM_BATCH_SIZE) once we
pass the reported article count, so progress keeps creeping forward monotonically
instead of freezing, and clamp to 99% so only the genuinely-final batch reports
100%.
This is a graceful heuristic, not exact progress (true accuracy would require a
pre-scan to count isArticleEntry matches up front); it removes the user-visible
"stuck at 99%" symptom with no change to batch semantics.
Closes#903
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Stored Files "partial stall" warning compares chunks in Qdrant against an
expected count from the ratio registry. The registry has an empty-pattern
catch-all (100 chunks/MB) that matches any filename, so a ZIM that matches no
specific pattern still gets a size-based estimate. For archives that are mostly
PDFs, images, or link-out stubs (e.g. irp.fas.org military-medicine), byte size
wildly over-predicts embeddable text: a 75 MB ZIM estimates ~7,236 chunks but
produces ~1, tripping a false "ingestion may have stalled" warning that re-embed
can't clear.
The catch-all is fine for rough aggregate disk-cost estimates, but it should not
drive a per-file stall signal. Add an `ignoreCatchAll` option to the ratio
lookup that excludes the empty-pattern row (returning null when only the
fallback would match), and use it in the warnings path so partial_stall only
fires when the registry has a *specific* expectation for the file. Files that
match a real pattern (wikipedia_, devdocs_, ifixit_, ...) are unaffected;
disk-cost/batch estimates keep using the fallback.
Closes#913
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The service update path stopped and renamed the old container aside before
the new one was confirmed running, but only wired up rollback if
createContainer threw or the 5s health check failed. A throw from
newContainer.start() itself (bad device/GPU config, host port already bound,
image incompatibility) bubbled straight to the outer catch, which returned a
generic 400 and never restored the old container, leaving the service down.
Retrying then wedged: the failed new container still held the service name, so
the next attempt's rename to `<name>_old` collided with the leftover from the
first attempt and threw the same error every time.
- Wrap newContainer.start() so a start failure removes the half-created
container and rolls back to the previous one.
- Clear any stale `<name>_old` before renaming so retries can't collide.
- Dedupe the three rollback sites into a single rollbackToOld() helper (also
removes a non-null assertion in the create-failure path that could itself
throw when no `_old` existed).
Refs #949
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When a service fails to install because something on the host already binds its
port, the user saw the raw dockerode error ("Bind for 0.0.0.0:11434 failed:
port is already allocated"), which is meaningless to a non-technical user. The
most common case is a native Ollama install holding 11434.
Add _humanizeDockerError() to map host port-conflict errors to an actionable
message that names the port and, for Ollama/11434, points at the likely cause
(a host Ollama service) with the commands to stop it. Unrecognized errors pass
through unchanged. Wired into the install failure broadcast and the thrown
error.
Closes#934
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a self-update path for the NOMAD admin/core image that runs without a
human in the loop, gated by layered safety checks. Recreation remains the
sidecar's job; this only decides *whether* to update now and requests it.
An hourly job (AutoUpdateJob) evaluates a side-effect-free decision pipeline
(AutoUpdateService.evaluate):
- opt-in: disabled by default
- in a user-configured local-time window (handles midnight wrap)
- an eligible release: same major (major bumps stay manual), strictly newer,
past a configurable cool-off, GA only (no drafts/prereleases), strict semver
- pre-flight: sidecar present, no in-flight system update / downloads / app
installs, and sufficient host disk (estimated from the registry manifest)
When all pass, it drives SystemUpdateService.requestUpdate() with a vetted tag.
Failure backoff auto-disables after 3 genuine update-request failures; transient
release-lookup errors are skips (offline-first appliances are routinely without
connectivity). Re-enabling clears the backoff state.
Settings UI exposes the toggle, window, and cool-off, with live status (eligible
target, in/out of window, last result/error). A `node ace auto-update:dry-run`
command exercises the full pipeline — and a deterministic --scenarios suite the
pure decision logic — without ever triggering an update.
New KVStore keys under autoUpdate.* hold config + last-run state.
The continuation dispatch in EmbedFileJob did not pass the running chunk
count forward, so each batch started with job.data.chunks undefined.
On the final batch, totalChunks collapsed to just that batch's result
and KbIngestState.markIndexed stored a value far below what Qdrant
actually held.
Add chunksSoFar to EmbedFileJobParams and thread it through the
continuation chain so the final markIndexed call reflects the true
total across all batches.
Closes#933
_applyHostStorageRoot only rewrote binds carrying the current
NOMAD_STORAGE_PATH and short-circuited when env == resolved root. That
left user-modified/custom apps — whose binds freeze the default prefix at
edit time — mounting an empty dir on the documented "set NOMAD_STORAGE_PATH
+ relocate the volume" path (#938).
- Match either the env value or the hardcoded default prefix; drop the
root == seededRoot no-op, excluding root from the candidates instead so
rewrites stay idempotent.
- Add ADMIN_STORAGE_DEST / DEFAULT_HOST_STORAGE_ROOT constants; replace the
cwd-derived storage dest so admin-mount lookup can't silently break.
- Comment why a transient inspect failure is deliberately not cached.
Child services (Kiwix, Ollama, Qdrant, Flatnotes, Kolibri) are created via the
Docker socket, so their bind mounts use a HOST path. That path was baked into
the services table at seed time from NOMAD_STORAGE_PATH (default
/opt/project-nomad/storage) — and NOMAD_STORAGE_PATH wasn't even present in
management_compose.yaml, so it always fell back to the default.
Result: relocating the admin storage volume in compose (e.g.
/mnt/big/storage:/app/storage) moved the admin's own data but left child apps
mounting the old, now-empty /opt/project-nomad/storage. Kiwix would come up
with no content. (#938)
- Add _resolveHostStorageRoot(): inspect the admin's own container, find the
bind backing /app/storage, and use its host Source as the single source of
truth (cached). Falls back to NOMAD_STORAGE_PATH/default if it can't be
inspected.
- Add _applyHostStorageRoot(): rewrite the host-side prefix of each storage bind
to that root. No-op when it already matches the seeded prefix, so default
installs are unaffected.
- Apply it in _createContainer (covers installs + dependency recursion) and in
the Kiwix library-mode recreate path.
- management_compose.yaml: add an explicit NOMAD_STORAGE_PATH knob and rewrite
the comments so the admin volume, env var, and disk-collector volume that must
agree are spelled out.
Closes#938
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add NOMAD-specific getting-started docs for all 9 curated Supply Depot apps,
the catalog/install fixes each one surfaced, and a way to reach the docs from
each app card.
Docs:
- New in-app Markdoc page admin/docs/supply-depot-apps.md covering all 9 apps
(Stirling PDF, File Browser, Calibre-Web, IT Tools, Excalidraw, Homebox,
Vaultwarden, Jellyfin, Meshtastic Web): first run/login, where data lives,
and offline behaviour. Registered in docs_service DOC_ORDER.
- Manage > Docs dropdown item linking each app to its section
(/docs/supply-depot-apps#<anchor>): anchor map in constants/supply_depot_docs.ts,
heading anchors via Markdoc {% #id %}, and hash-scroll on the docs page.
Install / catalog fixes:
- Stirling PDF: open straight to the tools (SECURITY_ENABLELOGIN=false; the old
v1 DOCKER_ENABLE_SECURITY flag was dead).
- File Browser: seed a known admin/nomad login (bcrypt) instead of a random
log-only password; scope visibility to content folders via mount selection and
move the DB out of the browsable root.
- Calibre-Web: bundle an empty Calibre library and seed it on install so setup
doesn't dead-end at db config (_runPreinstallActions__CalibreWeb).
- Homebox: swap the archived hay-kot image for the maintained sysadminsmedia fork.
- Vaultwarden: generate a self-signed cert on install and serve HTTPS by default
(_runPreinstallActions__Vaultwarden + ROCKET_TLS + ui_location https:8480), so
the web vault has the secure context it requires.
- Jellyfin: pre-create storage/media/{Movies,TV Shows,Music,Photos} so each
library points at its own subfolder, avoiding the overlapping-path issue that
silently hides content (_runPreinstallActions__Jellyfin).
- Seeder run() now also syncs ui_location for non-modified curated services, so a
catalog link/scheme/port change reaches existing installs on update.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a user-facing trigger to rebuild the Kiwix library index from the
ZIM files currently on disk. Covers the sideload case: a user copies a
.zim onto the box (USB, SSH, network share) outside NOMAD's download
flow, and Kiwix has no way to discover it without regenerating the
library index.
Reuses the existing native KiwixLibraryService.rebuildFromDisk() (which
also extracts embedded favicons natively), so no kiwix-tools container
and no icon-patch step are needed. In library mode (--monitorLibrary)
kiwix-serve hot-reloads the XML automatically; only legacy glob-mode
containers are restarted.
- KiwixLibraryService.rebuildFromDisk now returns the book count; adds
getBookCount() for the before/after delta
- ZimService.rescanLibrary() orchestrates rebuild + legacy restart
- POST /api/zim/rescan-library + ZimController.rescanLibrary
- "Rescan Library" button on the Content Manager page (shown when Kiwix
is installed) with a success toast reporting books found
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Users often saw the assistant disclaim ("Sorry, I couldn't find specific
context regarding X, but here's a general answer...") even when material
that directly answered the query was embedded. Two compounding causes,
both on the read/generation side:
1. The rag_context system prompt explicitly authorized the hedge: it told
the model to fall back to general knowledge and "acknowledge the
limitations," and mandated "According to the information available..."
citations that pushed small models into meta-commentary preambles.
Rewrote it to treat the retrieved context as the primary, authoritative
source, lead with the answer, fall back to general knowledge *silently*,
and never emit "couldn't find specific context" phrasing, while still allowing
the model itself to make some "sanity-check" decisions if the retrieved context
seems wildly incorrect or unrelated to the user's query.
2. The injected context labeled each block with a raw relevance score
(e.g. "Relevance: 42.3%"). nomic-embed cosine scores for genuinely
relevant passages sit ~0.4-0.6, so the number primed the model to
distrust correct context. Dropped the model-visible score (it stays in
the logs) and replaced it with a neutral source-title label.
Also added a conservative, score-scaled heading boost in rerankResults:
when query keywords match a chunk's section/article title (ZIM metadata we
already fetch), nudge its rank. Same diminishing-returns shape as the
existing boosts and gated behind the existing semantic-quality threshold,
so it can't promote a weak match.
Scope note: this removes the visible hedge and improves ranking of
already-retrieved chunks. It does NOT fix retrieval recall — diluted
1500-token chunks that never reach dense search's top-k are unaffected.
That's a follow-up (smaller chunks + BM25/RRF hybrid).
The Stored Files modal and per-file warnings panel both enumerate
distinct 'source' values in the qdrant content collection. The
existing implementation scrolls every point in the collection, 100 per
page, just to learn the unique sources. On a fully-ingested NOMAD this
is brutally slow: NOMAD3's 3.1M-point collection takes 50+ seconds to
return its ~40 distinct sources, and the same scan runs twice per modal
open (getStoredFiles + computeFileWarnings each pay it independently).
Qdrant 1.10+ supports a facet aggregation that returns the distinct
values of a payload field with their counts in a single call.
@qdrant/js-client-rest@1.16.2 (already pinned) exposes this as
`client.facet()`. Swap the three scroll loops:
- RagService.getStoredFiles
- RagService.computeFileWarnings
- RagService.\_scanAndSync (the source-set used for skip-already-embedded
decisions)
As a bonus, computeFileWarnings's per-source chunk count comes back
inline from the facet, so the secondary scroll that was incrementing a
counter is also gone.
`exact: true` everywhere because the counts feed Warning A's
zero_chunks decision and could mis-fire near thresholds otherwise. New
RagService.FACET_SOURCE_LIMIT = 10000 caps the facet response; real
NOMADs run ~100 sources max, this is comfortable headroom.
Expected impact on NOMAD3 (3M points, 40 sources): ~50s -> ~100ms per
endpoint. On an empty/fresh KB: no measurable change either way.
Replace literal string matching with ipaddr.js parsing
so equivalent encodings of 169.254.169.254
(::ffff:169.254.169.254, ::ffff:a9fe:a9fe,fully-expanded forms)
and fd00:ec2::254 are all rejected.
foreignKey/localKey were swapped on the ChatMessage → ChatSession
relation. Per Lucid's belongsTo contract, foreignKey is the column
on the child model and localKey is on the parent — so this must be
{ foreignKey: 'session_id', localKey: 'id' }, mirroring the inverse
hasMany on ChatSession.
The relation is not currently preloaded anywhere in the codebase, so
no runtime behavior changes today; this closes a latent bug that
would have broken any future preload('session') call.
RunDownloadJob's onComplete handler was unconditionally firing
EmbedFileJob.dispatch after every ZIM download, gated only by "is
Ollama installed?". The rag.defaultIngestPolicy KV setting was never
consulted, so users who explicitly set Auto-index to Manual still got
every newly-downloaded ZIM auto-embedded.
RagService.scanAndSync already handles Manual correctly by recording
pending_decision rows instead of dispatching (rag_service.ts:1587-1638
via decideScanAction). The post-download path skipped that gate.
Mirror the same check at the dispatch site: read the policy KV; if
Manual, firstOrCreate a pending_decision row in kb_ingest_state so the
per-file Index affordance from PR #909 surfaces the file the same way
scan-time-discovered Manual files do. firstOrCreate (not create) so a
re-download doesn't demote an existing indexed/failed row — the user
can explicitly re-index from the KB panel if they want fresh content.
Verified on NOMAD3: with rag.defaultIngestPolicy='Manual', every ZIM
downloaded today via Content Explorer (agriculture-essential +
computing-essential, ~62 MB across 7 files) wrote kb_ingest_state
rows with state='indexed' instead of pending_decision. Real bug,
not a hot-patch artifact.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Since PR #36b6d8e moved tier-installation tracking from a client-side
persistence model to a server-side derive-from-disk model, the card
display only ever updates once every file in a tier is fully on disk.
A user who picks Standard sees a blank card for the duration of the
download (often hours for large tiers like Wikibooks). Worse, if some
files finish before others, the card briefly shows a lower tier (e.g.
Essential) before promoting to the selected tier on completion, which
reads as "the system didn't accept my pick."
Backend: compute a sibling `downloadingTierSlug` by unioning installed
resource IDs with the IDs from active RunDownloadJob queue entries
(waiting + active + delayed, failed deliberately excluded), then
resolving the highest tier whose every resource is in that union. Set
only when it differs from `installedTierSlug` — no point reporting
"downloading Standard" when Standard is already fully installed.
Frontend: unify the prominent corner badge logic in CategoryCard to a
single `badgeTier` derived from selectedTier > downloadingTier >
installedTier. Spinner + "(downloading)" suffix when in flight,
checkmark for installed/selected. The pill row and lime border follow
the same source.
Verified on NOMAD3: backend correctly resolves the downloading tier
from in-flight BullMQ jobs; CategoryCard shows the spinner badge
immediately on Submit and switches to the checkmark variant when
downloads complete.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lift the hardcoded 'nomic-embed-text:v1.5' string out of both
RagService and OllamaService into a shared EMBEDDING_MODEL_NAME
constant in constants/ollama.ts. The duplicate in OllamaService
existed only to dodge a circular import with RagService; the
constants module has no service imports, so a shared constant
eliminates both the duplication and the drift risk called out
in the inline "keep in sync" comment.
Surfaces NOMAD's previously-silent model-stacking behavior and enforces a
"one chat model in VRAM at a time" invariant (the embedding model is
always exempt). Addresses Chris's NOMAD3 testing observation that
switching the dropdown in the chat header was invisibly slow on low-VRAM
hardware because the prior model was never unloaded — Ollama would
either evict it under memory pressure or load the new one on CPU after
the runner choked.
Three integration points all funnel through one new helper:
- **User changes the model dropdown** in an active chat session →
confirm modal "Switch to {newModel}? Switching to {newModel} will
start a new chat. Your current conversation stays available in the
sidebar." On confirm, fire `keep_alive: 0` against the previous chat
model, clear active session, set the new selection. Cancel snaps the
visible dropdown back to the previous value (no popup state leaks
into `selectedModel`).
- **User clicks a session in the sidebar** → no popup (system-initiated).
Restore the session's stored model into the dropdown and fire
`unloadChatModels(targetModel)` so anything that isn't the target
gets the unload hint.
- **Chat page first mount** → page-load normalization. Anything stacked
from a prior session gets the unload hint with the current selected
model as the target-to-preserve. Guarded by a ref so it only fires
once per page lifetime; gated on `selectedModel` being populated.
Backend surface is a single new helper and a single new route:
`OllamaService.unloadAllChatModelsExcept(targetModel: string | null)`
→ queries `/api/ps`, filters out (a) the embedding model name
(hardcoded `nomic-embed-text:v1.5` to avoid the RagService circular
import) and (b) `targetModel`, fires `POST /api/generate` with empty
prompt + `keep_alive: 0` in parallel against everything else.
Returns the names that were hinted. Best-effort: network or Ollama
errors are logged and swallowed so callers don't fail on housekeeping.
`POST /api/ollama/unload-chat-models` → thin wrapper validating
`{ targetModel?: string | null }`.
Why `keep_alive: 0` is safe against in-flight inference: per Ollama's
scheduler semantics, the hint sets the post-completion eviction timer
to zero — the runner is not terminated. If Session A is mid-response
on gemma when Session B fires the unload, gemma stays resident until
A's request completes, then evicts. The user-visible worst case is the
race where A's longer-running request re-extends the timer back to the
default and the unload is no-op'd; the next transition (or page reload)
gets another chance, and Ollama's own LRU catches up under memory
pressure regardless. Robust in-flight tracking deferred to a follow-up
if we see stale-state in the wild.
Base `rc`: v1.40.0 will inherit everything from rc.6 via the backmerge.
Frontend tests deferred to a follow-up PR; existing inertia tsconfig
errors are pre-existing and unrelated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Return a discriminated `EmbedSingleFileResult` from `RagService.embedSingleFile`
with `code: 'not_found' | 'inflight' | 'delete_failed' | 'dispatch_failed'` on
failure. `RagController.embedFile` now maps those codes to the correct status
instead of collapsing every failure to 409:
- not_found → 404
- inflight → 409
- delete_failed → 500
- dispatch_failed → 500
The `code` is also included in the JSON body so clients can branch without
string-matching `error`.
Closes the Manual-mode UX dead-end: after toggling 'Auto-index new content
for AI?' to Manual, a freshly-downloaded ZIM (or any pending_decision file)
had no UI path to opt in for embedding short of the global Sync Storage /
Re-embed All bulk actions. Per RFC #883 §5, each Stored Files row now
carries a state pill and an adaptive single-button action.
State pill (left of any existing warning chips):
- 'Indexed' — green; row had chunks in Qdrant or state row is 'indexed'
- 'Not Indexed' — neutral; state is pending_decision or browse_only
- 'Failed' — red
- 'Stalled' — amber
- admin_docs collapsed row has no pill ('Managed by NOMAD' carries it)
Adaptive action button (paired with the existing Delete button per row):
- pending_decision → 'Index' (force=false)
- browse_only → 'Index' (force=true)
- failed / stalled → 'Retry' (force=true)
- indexed + warning chip → 'Re-embed' (force=true; confirm modal first)
- indexed healthy / null → no action button (bulk Re-embed All covers it)
Backend: GET /api/rag/files now returns
{ files: Array<{ source, state, chunksEmbedded }> }
instead of a flat string[]. State + chunk-count come from a single
KbIngestState query unioned into the existing Qdrant-derived source list
(no new round trips). New POST /api/rag/files/embed validates the source is
known, refuses if any inflight job already targets the same filePath
(prevents double-click duplicate-chunk hazard), pre-deletes Qdrant points
when force=true, then dispatches via the existing _dispatchEmbedJobsFor
helper used by reembedAll.
Per-file Re-embed (force=true on an already-indexed file) routes through a
StyledModal confirmation since it deletes existing vectors before queueing
a fresh job — same destructive-action weight as Delete's inline confirm but
heavier since it affects search until the rebuild finishes.
Folds in PR #907's blank-screen fix because my new render needs the same
generic restored: `<StyledTable<KbFileGroup>>` and `record.displayName`
(instead of the unresolved `sourceToDisplayName(record.source)` that ships
in rc.5 and ReferenceErrors on modal open). PR #907 also adds title
tooltips on the three bulk-action buttons; those tooltips are NOT included
here — let PR #907 land first or independently for that part.
Multi-select bulk-opt-in deferred per discussion: most Manual-mode users
ingest 1-2 files at a time, the existing global toggle covers the bulk
case, and checkboxes would expand scope past what rc.6 should hold. Will
file a follow-up issue for an 'Index N pending files' single-click button
once this lands.
Tests-in-PR scope was limited to keeping `kb_file_grouping.spec.ts` green
after the StoredFileInfo[] signature change (added asInfos() wrapper).
Dedicated unit tests for embedSingleFile (unknown source / inflight refused
/ force=true delete-then-dispatch) and the new state-pill rendering will
land in a follow-up PR alongside Playwright coverage of the row actions.
Verification path: NOMAD3 currently runs project-nomad-admin:integration-
rc6-preview (PRs #907 + #908 atop rc.5). After this branch is built into a
new integration tag, I'll re-run targeted Playwright UAT on the KB modal
covering: state pill rendering per state, Index click on pending_decision
opts in cleanly, Retry on failed re-dispatches successfully, Re-embed
confirmation modal copy + delete-then-dispatch on the military-medicine
partial-stall row, and Delete flow untouched.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When a user opens AI Chat with content available but no global ingest
policy yet recorded, surface a one-time banner above the chat header
asking how they want new content handled:
- 'Index existing content' -> sets rag.defaultIngestPolicy=Always and
triggers a sync so pending_decision files queue immediately
- 'Maybe later' -> sets policy=Manual; existing and future content
waits in pending_decision until the user opts in from the KB modal
After either button is clicked the banner never reappears, because both
write the policy KV (the same one #894 manages via the KB modal toggle).
There is intentionally no 'dismiss without deciding' X — that would just
re-show the banner forever.
Backend
- New GET /api/rag/policy-prompt-state returns
{shouldPrompt, hasContent, totalFiles}
- RagService.getPolicyPromptState() reads KVStore('rag.defaultIngestPolicy')
and counts kb_ingest_state rows; shouldPrompt is true only when policy
is null AND scanner has seen >=1 file (avoids prompting on empty NOMADs)
Frontend
- New KbPolicyPromptBanner component (~120 LOC) handles the two-button
decision flow with optimistic loading state, success/error toasts, and
invalidates kbPolicyPromptState + ingestPolicy + embed-jobs + storedFiles
on success
- Mounted in components/chat/index.tsx as the first child of the main
content column so it sits above the chat title bar without taking space
when shouldPrompt is false (renders nothing)
- Reads aiAssistantName from Inertia page props so banner copy matches
the user's chosen assistant name
Stacks on feat/kb-policy-toggle (#894) because the policy KV mechanism
it writes through is introduced there. Both can land in rc.5; this PR
auto-rebases to rc once #894 merges.
Existing users on first upgrade to v1.32.0 will see this banner on first
chat visit post-upgrade — an explicit opt-in moment for content that was
already on disk. New users see it the first time they have curated
content downloaded.
Closes the 'zero_chunks warning has no row to attach to' gap surfaced by
the 2026-05-14 integration UAT. Before this fix RagService.getStoredFiles
returned only file paths that appeared in Qdrant's payload.source — so
files with 0 embedded chunks (video-only ZIMs, browse_only opt-outs,
ingestions that failed before producing any chunks) silently disappeared
from the KB panel's Stored Files table.
The fix unions the Qdrant scroll result with the disk-backed file paths
recorded in kb_ingest_state. Effect:
- lrnselfreliance_en_all_2025-12.zim (3.97 GB video-only ZIM, 0 chunks)
now appears in the table, picks up its zero_chunks warning chip
- Files in pending_decision under Manual policy show up so the user can
see what's waiting for opt-in
- Files in browse_only / failed states have a row for future per-card
Retry / Re-index actions (forthcoming, blocked on #886)
The state-machine query is wrapped in its own try/catch so a transient
DB error degrades to the Qdrant-only list rather than 500-ing the whole
panel — same defensive posture as the outer try/catch.
Stacks on feat/kb-ingest-state-machine (#888) because the union depends
on the kb_ingest_state table that PR introduces. Will rebase to rc once
#888 merges. Completes the second half of #895's warning surface; the
first half (partial_stall) already worked because those files have at
least some chunks in Qdrant.
`computeFileWarnings()` previously caught all errors and returned an empty
map, which the frontend rendered as "every file is healthy" — reintroducing
exactly the silent-failure mode this surface exists to expose.
Return `{ ok, warnings }`; flip `ok: false` from the catch. KB modal renders
an inline amber notice under the Stored Files header when `ok === false`,
leaving per-row warning rendering untouched. Transient failures self-heal on
the next 30s poll; no toast spam.
Surfaces two silent failure modes that the prior binary
"any-chunks-in-Qdrant ⇒ embedded" check could not distinguish from
healthy ingestion:
- **Warning A — Zero-chunk file** (file_size > 100 MB, chunks = 0)
Fires on video-only / image-only ZIMs (`lrnselfreliance_en_all`,
TED talks, etc.) that the pipeline completes "successfully" with no
extractable text. AI Assistant literally cannot reference these.
- **Warning B — Partial-embed stall** (chunks < 50% of expected from
the ratio registry). Surfaces the simple_wiki "266 of 600,000 chunks"
case observed during NOMAD1 ingestion testing — previously these
looked identical to fully-completed embeds in the UI.
Both warnings render only when their condition is met (silent by
default; noisy only on real problems).
Base is `feat/kb-ratio-registry` (#891) because Warning B's "expected
chunks" estimate comes from `KbRatioRegistry.estimateChunks()`. GitHub
fast-forwards to `rc` once #891 merges.
- `app/utils/kb_warning_decision.ts` — pure `decideWarnings(inputs)`
with thresholds (`100 MB`, `0.5×`) as exported constants. 10 unit
tests cover the healthy case, both warnings, the under/at/over
boundary, the registry-miss suppression, and the video-only registry
case (`expectedChunks: 0` correctly skips Warning B).
- `RagService.computeFileWarnings()` — single Qdrant scroll tallies
chunks per source, filesystem walk fills in zero-chunk files,
ratio registry estimates the expectation, decision function emits.
- New endpoint `GET /api/rag/file-warnings` returns
`Record<source, FileWarning[]>` (sources with no warnings are
omitted, so the frontend can `warnings[source] ?? []` for clean
defaults).
- KB modal: warnings render inline under the file name as amber-tinted
pills. Polled every 30s alongside the existing health check.
- Warning C — chunks skipped due to length. PR #890 (#881 fix) prevents
the silent drop at the embed boundary, so the underlying condition
shouldn't fire anymore. If we still want to surface "we truncated
N chunks to fit", that needs separate `skipped_count` tracking in
EmbedFileJob — a Phase 2 follow-up.
- Suppressing Warning B during active mid-ingestion. The user can cross-
reference the Processing Queue to know it's in-flight; suppressing
warnings while a job runs would mask real stalls where the job died
mid-batch. Will revisit when per-card status is wired through.
- Use of `kb_ingest_state.chunks_embedded` (#888) as the chunk count
source. This PR uses Qdrant scroll directly so it can land
independently of #888.
- 10 new unit tests on `decideWarnings`, all pass
- Type-check clean
- Hot-patch + browser smoke test deferred until #891 lands (the ratio
registry needs to exist in the DB for `estimateChunks()` to return
non-null estimates — without it, only Warning A fires which is still
useful but Warning B stays dormant)
When a user picks a tier in TierSelectionModal, show how much additional
disk space the AI Assistant will need if the new ZIMs are indexed, plus
a policy-aware footer explaining whether they'll auto-index (Always) or
wait for opt-in (Manual). Estimates consume #891's KbRatioRegistry via a
new POST /api/rag/estimate-batch endpoint.
Backend
- New POST /api/rag/estimate-batch route + RagController.estimateBatch
- VineJS schema accepting array of {filename, sizeBytes}, capped at 500
- KbRatioRegistry.estimateBatch aggregates via the existing prefix-match
lookup, returns {totalChunks, totalBytes, hasUnknown}
- New BYTES_PER_CHUNK_ON_DISK constant (~8 KB: 3 KB vector + ~3 KB chunk
text + ~2 KB payload/index overhead). Tunable; will be replaced by
Phase 4 self-calibration once we have real measurements.
- Controller normalizes incoming filenames via path.basename so callers
that send full paths or URLs still match registry prefixes correctly.
Frontend
- api.estimateEmbeddingBatch() client method
- TierSelectionModal: when localSelectedSlug is set, resolve the tier's
resources (incl. inherited tiers), POST to /estimate-batch, and render
a new info block with the +~X GB figure + ingest-policy copy. Also
fetches rag.defaultIngestPolicy so the same block surfaces whether
indexing will fire automatically or wait for the user.
- resourceFilename() helper extracts the basename from the resource URL
so the registry lookup hits the right prefix regardless of mirror.
Tests
- 4 new cases in tests/unit/kb_ratio_lookup.spec.ts covering the
estimateBatch aggregator: standard sum, unknown-flagging, video-only
ZIM (0 chunks but known, hasUnknown stays false), empty input.
Stacks on feat/kb-ratio-registry (#891) — consumes the registry table
seeded by that PR. Once #891 merges to rc, this PR auto-rebases.
Out of scope for this PR (deferred to follow-ups):
- Per-batch opt-in checkbox (RFC §1's '☑ Also index these for AI') needs
a per-batch policy override path and is a separate PR
- Guardrail modal at 50 GB / 10% free / 6 hr thresholds (RFC §7) is also
separate; this PR is informational, not gating
- Time-to-embed estimate awaits a chunks-per-second metric per host
* feat(KB): per-file ingest state machine (Phase 1 of RFC #883)
Adds a persistent state machine for AI knowledge-base ingestion so the
scanner can distinguish "fully indexed", "user opted out", "failed", and
"stalled" from each other — none of which were derivable from the prior
binary "any chunks in Qdrant ⇒ embedded" check.
## What lands
- New table `kb_ingest_state` keyed by `file_path` with enum state column
(`pending_decision | indexed | browse_only | failed | stalled`).
Independent of `installed_resources` so it covers both curated downloads
and manually-uploaded KB files.
- New KV key `rag.defaultIngestPolicy` (string: `Always | Manual`).
Registered now but not consumed yet — JIT prompt + wizard step land in
Phase 3 of the RFC.
- `EmbedFileJob.handle` writes state on terminal outcomes:
- Success (final batch) → `indexed` + chunks count
- `UnrecoverableError` → `failed` + error message
- Retryable errors are left to BullMQ's existing retry path
- `scanAndSyncStorage` swaps the binary qdrant check for a state-aware
decision tree (see `decideScanAction`). Existing installs auto-backfill
on first scan: files with chunks in Qdrant but no state row become
`indexed`; new files start as `pending_decision`.
- `deleteFileBySource` drops the state row last, so removed files
disappear entirely instead of leaving an orphan that the next scan
would re-dispatch into nothing.
## What does NOT land here
- Ratio registry (separate PR) — needed for partial-stall detection and
cost estimates, but a separable concern.
- #880 follow-up initial-progress anchor (separate tiny PR).
- Phase 2 UI (status pill, per-card actions, conditional warnings).
- Phase 3 policy surfaces (wizard step, JIT prompt, guardrail modal).
- PR #886's bulk-action hookup — `_deletePointsBySource` / Re-embed All
/ Reset & Rebuild would also want to set state, but #886 isn't merged
yet; that wiring goes in a follow-up once #886 lands.
## Target
This is forward work for v1.40.0 (RFC #883). Branching off `rc` because
that's the current latest base and post-GA Jake will sync rc→dev; a
retarget at PR-open time is a fast-forward if requested.
## Tests
- 9 new unit tests for `decideScanAction` covering all five states plus
the no-row / chunks-present / chunks-missing combinations
- Type-check clean
- Smoke-tested end-to-end on NOMAD3 via hot-patch:
- Backfill: 5 ZIMs + 2 KB uploads with existing chunks in Qdrant all
came back `indexed` on first scan
- Pending dispatch: a video-only ZIM with no chunks (`lrnselfreliance`)
came back `pending_decision` and was correctly re-dispatched (Bull
deduped to its historical `:completed` jobId — bgauger's #886 fix
drains that)
- Delete hook: deleting a KB upload via `DELETE /api/rag/files`
removed both the disk file and the state row
* feat(KB): Always/Manual ingest policy toggle (RFC #883 §1/§4)
Activates the `rag.defaultIngestPolicy` KV registered in Phase 1
(#888) so users on a fresh install (or anyone who picks Manual mode)
no longer get every new ZIM auto-dispatched to the embed pipeline.
## Stacks on #888
This PR's base is `feat/kb-ingest-state-machine` (#888). The state
machine has to be in place for the decision function to be policy-aware;
GitHub will fast-forward the base to `rc` once #888 merges.
## Backend changes
- `decideScanAction` now takes a `policy: 'Always' | 'Manual'` argument
(defaults to `Always` for backward compatibility).
- New `ScanAction` kind: `create_pending`. Manual mode records that the
scanner has seen a new file (so the UI can surface a per-card Index
affordance later) without dispatching an EmbedFileJob.
- `scanAndSyncStorage` reads the KV and passes it through. The scan-result
log line now includes the active policy and a `waiting on user` count
for Manual-mode hits.
- `rag.defaultIngestPolicy` added to `SETTINGS_KEYS` so it's reachable
through the existing `GET/PATCH /api/system/settings` surface — no new
endpoint.
## Frontend changes
- New section in the KB panel between "Why upload" and "Processing Queue":
"Auto-index new content for AI? [Always | Manual]" — segmented radio
with copy explaining the 5-10× disk multiplier. Default Always.
- `useQuery('ingestPolicy')` reads the current value; clicking the
inactive option mutates and shows a notification confirming the new
behavior.
## Tests
- 14 unit tests on `decideScanAction` (was 9) — split into Always-mode
cases (preserves Phase 1's contract) and Manual-mode cases
(`create_pending`, `pending_decision → skip`, etc.).
- Type-check clean.
- Hot-patch + browser verification deferred until #888 lands; the state
machine smoke-tested cleanly on NOMAD3 in #888's PR, and this PR's
decision-tree changes are exhaustively unit-tested.
## RFC open question §3 — policy-change re-trigger
Switching Manual → Always doesn't auto-dispatch existing `pending_decision`
rows immediately. The next scan re-evaluates and dispatches them under
the new policy. This matches the RFC's "treat the switch as I've-
thought-about-it" instinct for the guardrail; full guardrail
implementation lands in Phase 3 task 14.
---------
Co-authored-by: Jake Turner <52841588+jakeaturner@users.noreply.github.com>
Each in-flight (or stuck) embedding job gets a colored health pill,
relative-activity timestamp, and chunk counter so users can tell at a
glance whether ingestion is making progress.
## Health states
- **🟢 Active** — last batch < 2 min ago
- **🟡 Slow** — last batch 2-5 min ago (CPU-paced multi-batch ingestion
lives here naturally; not always a problem)
- **🔴 Stalled** — last batch > 5 min ago (likely real problem)
- **⚪ Waiting** — queued, no batch started yet
- **🔴 Failed** — job recorded failed status
## What lands
- New backend util `kb_job_health.ts` with pure `computeJobHealth(input)`
decision function. Time-based thresholds (2 min / 5 min) inlined as
constants. 9 unit tests pin the boundaries.
- `EmbedJobWithProgress` gains `lastBatchAt`, `startedAt`, `chunks` —
already set by `EmbedFileJob.handle` on every batch transition, just
not previously surfaced through `listActiveJobs`.
- Frontend `kb_job_health_display.ts` maps each status to a Tailwind
dot color, label, and aria-label so backend and UI stay in sync.
- `ActiveEmbedJobs.tsx` renders the pill, "last activity Xs ago", and
chunk counter above each progress bar. Adds a manual Refresh button
and "Last updated Xs ago" line — the existing 2s/30s auto-poll
cadence in `useEmbedJobs` is left intact.
- Live tick at 5s keeps the relative timestamps current without
re-fetching from the API.
## Not in scope
- Per-card Cancel / Retry / Un-index — separate Phase 2 PR
- Conditional warnings A/B/C — separate Phase 2 PR
- Computing throughput rate (chunks/min) — needs ratio registry consumer
(Phase 2 follow-up); for now the pill answers the "is it stuck?"
question directly without a rate estimate.
Foundation for the cost estimates and partial-stall detection that
Phase 2 will surface. No consumers yet — this PR just lays the table,
the seed rows, and the lookup helper so subsequent UI work has
estimates available without a per-ZIM benchmark.
## What lands
- New table `kb_ratio_registry` (pattern, chunks_per_mb, sample_count,
notes). Migration creates and seeds heuristic defaults from the RFC
appendix: devdocs (1100/MB), Wikipedia variants (270/MB), iFixit
(50/MB), Stack Exchange Q&A (200/MB), video-only ZIMs (0), plus a
catch-all fallback at 100/MB.
- `KbRatioRegistry` model with static `lookup()` and `estimateChunks()`.
- Pure helper `kb_ratio_lookup.ts` doing longest-prefix-match — a
specific entry (`wikipedia_en_simple_`) overrides a broader one
(`wikipedia_en_`). 9 unit tests covering the lookup boundary.
- `sample_count` starts at 0 (heuristic seed) and is reserved for
Phase 4 self-calibration to increment as observed ZIMs update each row.
## Not in scope
- Self-calibration on successful ingestion (Phase 4)
- UI consumers — Warning B (partial-embed stall) and the storage budget
meter / time estimates land in Phase 2.
## Tested
- Type-check clean
- 9 unit tests pass for `findChunksPerMb` and `estimateChunkCount`
- Migration applied on NOMAD3 via hot-patch; 9 seed rows verified in DB
The OpenAI-compatible /v1/embeddings fallback path can't pass
`truncate:true` / `num_ctx:8192` to the model, so any chunk that
exceeds the model's loaded context_length (often 2048 for
nomic-embed-text:v1.5) returns a 400 BadRequestError and is silently
dropped from Qdrant. Two CPU-only ingestion runs on NOMAD1 hit this
on dense technical content (medlineplus, arduino.stackexchange) even
after PR #763's num_ctx fix on the native path.
Pre-cap each input string at 4000 chars before either backend call.
That's ~1000-2000 tokens depending on density, comfortably under the
model's 2048 default. The chunker in RagService is sized for
MAX_SAFE_TOKENS=1600 (3200 chars at its conservative 2 chars/token
estimate), so well-formed inputs are never touched; this is purely a
runtime safety net for the edge cases that slip through.
Also stop swallowing the original error in the catch. The bare
`} catch {}` here has masked recurring "input length exceeds context
length" failures for months (#369, #670, #881). Capture and warn-log
the message so future investigations see why we fell back.
Same root cause as #369 and #670 which were closed without an actual
fix to the fallback path.
Each continuation batch of a multi-batch ZIM embed runs as a fresh
BullMQ job, so handle() ran the hardcoded `safeUpdateProgress(job, 5)`
even when the file was already 100k articles into a 600k-article ZIM.
The UI gauge briefly dropped to 5% before the per-batch onProgress
callback caught up to the true overall percentage, reading as a
backward jump every time a new batch started.
Compute initialPercent from batchOffset / totalArticles when available,
falling back to 5 for single-batch files (uploaded PDFs, txts) where
totalArticles isn't set. Capped at 99 to leave headroom for the 100%
final-batch marker.
Follow-up to PR #880 (which fixed the 0-100% scaling during a batch
but still had the initial-frame regression).
Adds a persistent state machine for AI knowledge-base ingestion so the
scanner can distinguish "fully indexed", "user opted out", "failed", and
"stalled" from each other — none of which were derivable from the prior
binary "any chunks in Qdrant ⇒ embedded" check.
## What lands
- New table `kb_ingest_state` keyed by `file_path` with enum state column
(`pending_decision | indexed | browse_only | failed | stalled`).
Independent of `installed_resources` so it covers both curated downloads
and manually-uploaded KB files.
- New KV key `rag.defaultIngestPolicy` (string: `Always | Manual`).
Registered now but not consumed yet — JIT prompt + wizard step land in
Phase 3 of the RFC.
- `EmbedFileJob.handle` writes state on terminal outcomes:
- Success (final batch) → `indexed` + chunks count
- `UnrecoverableError` → `failed` + error message
- Retryable errors are left to BullMQ's existing retry path
- `scanAndSyncStorage` swaps the binary qdrant check for a state-aware
decision tree (see `decideScanAction`). Existing installs auto-backfill
on first scan: files with chunks in Qdrant but no state row become
`indexed`; new files start as `pending_decision`.
- `deleteFileBySource` drops the state row last, so removed files
disappear entirely instead of leaving an orphan that the next scan
would re-dispatch into nothing.
## What does NOT land here
- Ratio registry (separate PR) — needed for partial-stall detection and
cost estimates, but a separable concern.
- #880 follow-up initial-progress anchor (separate tiny PR).
- Phase 2 UI (status pill, per-card actions, conditional warnings).
- Phase 3 policy surfaces (wizard step, JIT prompt, guardrail modal).
- PR #886's bulk-action hookup — `_deletePointsBySource` / Re-embed All
/ Reset & Rebuild would also want to set state, but #886 isn't merged
yet; that wiring goes in a follow-up once #886 lands.
## Target
This is forward work for v1.40.0 (RFC #883). Branching off `rc` because
that's the current latest base and post-GA Jake will sync rc→dev; a
retarget at PR-open time is a fast-forward if requested.
## Tests
- 9 new unit tests for `decideScanAction` covering all five states plus
the no-row / chunks-present / chunks-missing combinations
- Type-check clean
- Smoke-tested end-to-end on NOMAD3 via hot-patch:
- Backfill: 5 ZIMs + 2 KB uploads with existing chunks in Qdrant all
came back `indexed` on first scan
- Pending dispatch: a video-only ZIM with no chunks (`lrnselfreliance`)
came back `pending_decision` and was correctly re-dispatched (Bull
deduped to its historical `:completed` jobId — bgauger's #886 fix
drains that)
- Delete hook: deleting a KB upload via `DELETE /api/rag/files`
removed both the disk file and the state row
Co-authored-by: Jake Turner <52841588+jakeaturner@users.noreply.github.com>
onWikipediaDownloadComplete was deleting every file whose name starts
with `wikipedia_en_`, treating distinct corpora (simple, medicine,
wikivoyage, climate_change, etc.) as competing versions of the same
selection slot. Whichever wiki finished second silently wiped the
other from disk.
Match by filename stem instead — strip the trailing `_YYYY-MM(-DD).zim`
date suffix and only delete files with the same stem as the new
download. Different release dates of the same variant still get cleaned
up; distinct variants are preserved.
Extracted the predicate to `app/utils/zim_filename.ts` so the boundary
is covered by unit tests (8 cases incl. the #884 repro scenario).
Before this change, the Active Downloads / Processing Queue UI showed the
ingestion progress gauge jumping wildly during multi-batch ZIM ingestion
(e.g. 5% → 88% → 27% → 5% → 56% → 36% over ~60 seconds for cooking SE).
Each continuation batch is a separate BullMQ job, and `EmbedFileJob.handle()`
reported `job.progress` in two different reference frames depending on
where it was in the batch lifecycle:
- During-batch (via the onProgress callback): 5% → 95% scaled across
"% through this batch's chunks"
- End-of-batch (just before dispatching the next): overwritten to
`(nextOffset / totalArticles) * 100` — % through the whole file
- Next continuation batch starts with progress = 5% explicitly, then
climbs through the per-batch range again
`listActiveJobs()` returns the latest active BullMQ job's progress. With
GPU-accelerated ingestion completing a batch every ~4 seconds, the UI
saw the jobId rotate constantly and the gauge whipsaw between the two
reference frames.
`totalArticles` was already wired through the EmbedFileJob params shape
and used end-of-batch — but RagService never actually populated it,
so any frame-scaling that depended on it silently fell back to the
per-batch range. Two fixes together:
1. `ZIMExtractionService.extractZIMContent()` now returns
`{ chunks: ZIMContentChunk[]; totalArticles: number }` instead of a
raw chunks array, surfacing `archive.articleCount` to the caller.
Single caller (rag_service) updated to destructure.
2. `RagService.processZimFile()` includes `totalArticles` in its result
so `EmbedFileJob.dispatch()` can propagate it to the continuation
batch (which the existing code already does via
`totalArticles: totalArticles || result.totalArticles`).
3. `EmbedFileJob`'s onProgress callback scales the service-reported
per-batch percent into the overall-file frame when `totalArticles`
is known: `((batchOffset + (percent/100) * ZIM_BATCH_SIZE) /
totalArticles) * 100`. Capped at 99% to leave room for the explicit
100% set at file completion. Falls back to the original 5-95% range
for single-batch files (uploaded PDFs/txts) where totalArticles is
undefined — the gauge then represents % through the only batch,
which is what the UI expects for one-shot files.
Validated on NOMAD8 (RX 6800, ROCm-accelerated nomic):
- devdocs python (small, ~1500 articles): batch progressions seen
monotonically across continuation jobIds:
1501@30% → 1510@33% → 1514@43% → 1518@52%.
- ifixit (huge, ~100k articles): stays near 3% for the first many
batches at offset 0..3000 — correct, the file is enormous.
- wikipedia_en_medicine (large, ~70k articles): stays near 0-1% for
the first batches — also correct.
- Brief 0-5% blip on continuation handoff (the explicit
`safeUpdateProgress(job, 5)` at batch start, before the first
onProgress callback fires) — visible but quickly resolves to the
overall-frame value. No more 5% ↔ 88% chaos.
Jake noted that `inspect.State.StartedAt` could be missing/malformed,
which would land NaN inside `container.logs({ since, until })`. Add
defensive validation that the parsed timestamp is finite and positive
before using it, with a fallback to the previous tail:500 strategy
(plus a warn log) when it isn't. Happy path is unchanged.
Two related fixes to make the System Information page reliably show real
GPU info instead of misleading lspci BAR0 readings or N/A.
1. Generalize bogus-VRAM detection to AMD.
Same root cause as #835 (NVIDIA showing 32 MB), this time for AMD: lspci
parses the first PCI memory Region (BAR0, typically 1-16 MiB on Navi
cards) as `vram`. On NOMAD8 (Threadripper 3960X + Radeon RX 6800), the
System Information page showed "1 MB" instead of "16 GB". PR #850 fixed
this for NVIDIA by clearing the bogus value and re-running the Ollama
log probe; the check was vendor-gated to NVIDIA only.
`isBogusNvidiaVram` becomes `isBogusDgpuVram` with a `isDiscreteGpuVendor`
helper matching /nvidia|advanced micro devices|amd|ati/i. Same 256-MiB
threshold — no real discrete GPU has less than that, while Intel iGPUs
(which legitimately report small shared-memory VRAM via lspci) are left
untouched. The probe gate condition is similarly renamed.
2. Read Ollama logs from the startup window, not tail:N.
`getOllamaInferenceComputeFromLogs()` was reading the last 500 log lines
and grepping for the "inference compute" line. That line is written once
during Ollama's GPU discovery phase within seconds of startup. Under
active embedding workloads we measured >1000 log lines/min, which pushes
the line past any reasonable tail within minutes — at which point the
probe returns null and the UI flips to "GPU Not Accessible" even though
Ollama is happily using the GPU (size_vram > 0 in /api/ps).
Switch from `tail: 500` to `since: containerStartedAt, until:
containerStartedAt + 300s`. The 5-minute window is bounded regardless of
container uptime and always captures Ollama's GPU discovery output. The
inference-compute line is emitted in the first few seconds of startup, so
5 min is generous headroom.
Validated on NOMAD8 (RX 6800, container uptime ~10 min with sustained
ingestion that generated 6,345 log lines):
Before:
controllers[0]: { model: "Navi 21 ...", vram: 1 }
After (bogus AMD VRAM cleared, log probe stale due to tail:500 churn):
controllers[0]: { model: "Navi 21 ...", vram: null }
gpuHealth: { status: "passthrough_failed" }
-> UI shows "N/A" and the banner from PR #208
After (bogus cleared + log probe reads startup window):
controllers[0]: { model: "AMD Radeon RX 6800", vram: 16384 }
gpuHealth: { status: "ok", hasRocmRuntime: true, ollamaGpuAccessible: true }
-> UI shows "16 GB", no banner
Both branches of the fix exercise correctly: NVIDIA path unchanged
(same code, just renamed identifiers), AMD path now triggers the probe
and the probe reliably finds the GPU info regardless of container age.
Stacks on top of the multi-batch ZIM ingestion fix. After that fix,
multi-batch ZIM ingestion completes correctly — but on installs where
Ollama runs the embedding model on CPU (currently every AMD ROCm
install, since Ollama's ROCm build doesn't accelerate nomic-bert),
the now-correct sustained 100% CPU saturation across all cores can
starve other services hard enough to take the box down. Confirmed
on a Threadripper 3960X + RX 6800 NOMAD: a wikipedia-class ZIM
ingestion pegged 48 threads cleanly enough that sshd lost
banner-exchange responsiveness and the box ultimately required a
power-cycle.
NVIDIA installs aren't affected — nomic-embed-text:v1.5 runs at
100% GPU on RTX 5060 (verified via `ollama ps`).
Detect placement at runtime, pace only when needed:
1. OllamaService.isEmbeddingGpuAccelerated() — queries /api/ps and
returns true if any loaded embedding model reports size_vram > 0.
Fails closed (returns false) if /api/ps is unreachable or no embed
model is loaded yet — over-pacing is safer than crashing.
2. EmbedFileJob.handle() — between batches (hasMoreBatches: true
branch), check placement and `await setTimeout(CPU_BATCH_DELAY_MS)`
when CPU-only. CPU_BATCH_DELAY_MS = 1000 (1s) — enough to give the
OS scheduler a window for sshd/disk-collector/etc., small enough
that total ingestion time isn't meaningfully affected (each batch
is ~60-90s of work).
GPU-accelerated installs see zero behavior change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Every static call site instantiated a fresh QueueService (24 call sites
across 8 files). QueueService.getQueue() opens a BullMQ Queue per call
when not cached, and each Queue opens two ioredis connections (one for
commands, one blocking). Because every static call constructed a new
QueueService, its internal `queues` cache was never shared, every call
opened a fresh pair, and none were ever closed.
In normal operation this leaked a few connections per API hit. During
multi-batch ZIM ingestion after PR #872 (where EmbedFileJob.handle()
dispatches the next batch every 50 articles), every batch completion
opened two new connections. On NOMAD3 at ~one batch every 4s sustained,
that's ~1800 leaked connections/hour. Redis hit its 10,000-maxclient
ceiling in ~5 hours and the admin container fell into an EPIPE flood
that required a restart to recover.
Fix: collapse QueueService to a true process-wide singleton with a
private constructor and getInstance() accessor. The existing per-queue
Map is now shared across every dispatch / status / cleanup call, so each
queue's underlying connections are opened exactly once for the lifetime
of the process. close() now clears the map so the singleton can be torn
down cleanly if a graceful-shutdown hook is ever wired up.
Validated on NOMAD3 (RTX 5060, v1.32.0-rc.4 + this patch hot-applied):
under sustained multi-batch wikipedia_en_simple_all_nopic ingestion,
connected_clients held flat at 21-22 across a 5-minute window. Pre-fix
the same scenario climbed to 10,000+ over hours.
EmbedFileJob.dispatch() uses a deterministic per-file jobId
(sha256(filePath).slice(0,16)) for every batch. The parent batch's
handle() calls EmbedFileJob.dispatch({ batchOffset }) before returning,
so the parent is still in `active` state and locked when the
continuation tries to enqueue. BullMQ silently returns the locked
parent instead of creating a new job — and in newer BullMQ versions
it does so without throwing, so the existing
`catch (error.message.includes('job already exists'))` branch never
fires. After the parent completes, its entry stays in the `completed`
ZSET (held by `removeOnComplete: { count: 50 }`), continuing to trip
jobId dedupe for any subsequent re-dispatch attempts.
Result: every NOMAD install since 2026-02-08 (feat: zim content
embedding) with a multi-batch ZIM (wikipedia, cooking SE, ifixit,
lrnselfreliance, etc.) has only the first 50 articles indexed in
qdrant. The RAG feature has been silently degraded for ~3 months —
the user sees the file appear in their KB, qdrant accumulates ~50
articles' worth of vectors, and pagination quietly halts. No error
surfaces anywhere.
Fix: dispatch() skips the deterministic jobId for continuation batches
(batchOffset > 0), letting BullMQ auto-generate a unique one so each
batch stacks as an independent queue entry. Initial dispatches keep
the deterministic jobId so re-triggering an install (UI re-click,
sync rescan) remains idempotent. The existing 'job already exists'
branch is now gated on !isContinuation, since by construction
continuation batches will never hit dedupe.
Validated on NOMAD8 (RX 6800 / Threadripper 3960X, rc.3 + this patch):
devdocs_en_python (~1,500 chunks across multiple batches) correctly
paginates end-to-end. admin.log shows the expected sequence of
"Dispatched embedding job for file: X (continuation @ offset N)"
followed by "Starting embedding process for: X (batch offset: N)"
for each batch.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The short-conversation skip in `rewriteQueryWithContext` used `userMessages.length <= 2`,
which short-circuits both the very first turn AND the first follow-up. The follow-up is
the moment the rewriter matters most — it's where pronouns and shorthand ("the bars",
"how long does it last?") need to be resolved against earlier turns before the embedding
search runs. With the rewriter skipped, RAG queries against the raw last message, scores
nothing above the 0.3 threshold, and no context gets injected for that turn.
The visible symptom is the assistant treating the first follow-up in any chat as a
brand-new question — e.g. "great - they threw up 2 of the bars it looks like" answered
as if it were a recipe-bars question, with no carry-forward of the prior chocolate-
poisoning context.
Threshold lowered to `< 2`: skip only when there's exactly one user message (nothing to
rewrite from). From the first follow-up onward the rewriter runs, as originally intended
before commit 96e5027.
Validated against `mistral-nemo:12b` on NOMAD3 by hot-patching the compiled controller
and replaying the dog-chocolate scenario. Post-patch response correctly threads "3
Hershey's bars" from turn 1 into turn 2's answer; pre-patch (per reporter's screenshot)
pivoted to peanut butter bar recipes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
download.kiwix.org (and some of its mirrors) don't always set a
Content-Type header on .zim responses. The MIME validator was reading
`headers['content-type'] || ''`, then running each allowlist entry
through `''.includes(...)` which is always false, so every download
from those hosts was rejected with `MIME type is not allowed`.
RFC 7231 §3.1.1.5 says missing Content-Type may be treated as
application/octet-stream by the recipient, and that's already in every
binary-content allowlist we use (ZIM, PMTILES, base assets). Default
the missing case to that and the validator does the right thing.
Strict callers that don't list octet-stream still reject as before.
Closes#855.
PR #804's AMD branch in `updateContainer()` overrode `newImage` to
`ollama/ollama:rocm` and then persisted that literal string to
`service.container_image` (line 1273). Two downstream consequences for
every AMD user who clicked Update on AI Assistant:
1. Apps page (`apps.tsx`) extracts the displayed version from
`container_image` and rendered the literal string "rocm".
2. `ContainerRegistryService.getAvailableUpdates()` parsed `currentTag =
"rocm"`, which isn't semver, so `parseMajorVersion` returned NaN, the
filter didn't reject newer tags by major-version, and `isNewerVersion`
treated any future tag as newer. Result: the same update reappeared
on every check, forever.
Fix: separate "what we run" from "what we persist". `runtimeImage`
holds the tag passed to `docker.pull()` and `createContainer()` (still
`:rocm` for AMD), while `newImage` keeps the semver tag and is the
value written to the DB. Surgical: 3 references renamed plus 1
declaration added.
The install path (`_createContainer`) already had the right shape
(runtime-only override, no DB write of the override), so this PR only
touches `updateContainer`.
Test plan:
- `npm run typecheck` passes locally.
- Manual repro on NOMAD2 (AMD HX 370 / 890M, rc.2): before fix, DB
shows `container_image = ollama/ollama:rocm` after triggering an
Ollama update via Settings > Apps; Apps page shows version "rocm";
`/api/system/services/check-updates` immediately re-reports the same
update available. After fix, DB shows `container_image =
ollama/ollama:<targetVersion>`; Apps page shows the semver; check-
updates does not re-report the same update.
- nomad_ollama container itself still runs the `:rocm` image
(verified via `docker inspect`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR #804 added pciutils to the admin image so AMD detection could fall
back to lspci. Side effect for NVIDIA: si.graphics() now finds the card
via lspci but reads the BAR0 region size (16-32 MiB on most NVIDIA
cards) as VRAM, since nvidia-smi isn't installed in the admin image to
enrich the result. A GTX 1050 Ti showed 32 MB instead of 4096.
The nvidia-smi-via-Ollama and Ollama log probes already give the right
number, but they only ran when graphics.controllers came back empty.
Extend the trigger so they also run when the only entries are NVIDIA
controllers reporting under 256 MiB (no real dGPU has that little). If
the probes can't reach a value either (Ollama not installed, passthrough
broken), VRAM now falls back to N/A instead of the bogus 32.
Verified locally on an RTX 4070 Ti by simulating the container condition
(lspci available, nvidia-smi unreachable). Before the fix: vram 32, model
"AD104 [GeForce RTX 4070 Ti]". After: vram 12288, model "NVIDIA GeForce
RTX 4070 Ti", from the Ollama inference-compute log line. Also confirmed
the same result inside the actual admin Docker image.
Closes#810.
## Bug A: HSA_OVERRIDE_GFX_VERSION=11.0.0 was unconditional
PR #804 set HSA_OVERRIDE_GFX_VERSION=11.0.0 for any AMD GPU. The inline comment
claimed this was harmless on supported discrete cards (gfx1030 RX 6800, etc.) — empirically false. With the override, Ollama crashes during GPU discovery on
gfx1030 and falls back to CPU silently. Affects every NOMAD user with an
RX 6800 or other RDNA 2 discrete card.
The correct value depends on the gfx version:
- gfx1030, gfx1100, gfx1101, gfx1102: officially supported by ROCm — no override
- gfx1031..gfx1036 (RDNA 2 variants + iGPUs like Rembrandt 680M): 10.3.0
- gfx1103, gfx1150, gfx1151 (Phoenix 780M, Strix 890M, Strix Halo): 11.0.0
### Resolution chain in `_resolveAmdHsaOverride()`
1. KV `ai.amdHsaOverride` — manual override; accepts 'none' to disable, or a
semver-style value to force.
2. Marker file `/app/storage/.nomad-amd-gfx` — written by install_nomad.sh
based on lspci codename. Mapped to override via `_mapGfxToHsaOverride()`.
3. Default: `11.0.0` — preserves prior behavior so existing iGPU users
(780M / 890M, the dominant AMD population today) don't regress on upgrade.
Discrete RDNA 2 users on existing installs can opt out via
`ai.amdHsaOverride='none'` and force-reinstall AI Assistant, OR re-run
install_nomad.sh to refresh the marker file.
The helper is used in both `createContainer` (initial install) and
`updateContainer` (image update) paths, replacing the unconditional push.
## Bug B: BenchmarkService had no AMD discrete detection path
`BenchmarkService.getHardwareInfo()` had three GPU detection fallbacks:
1. `si.graphics()` — empty inside Docker for AMD
2. nvidia-smi — NVIDIA only
3. AMD APU regex from CPU model — integrated only
Result: AMD discrete cards (RX 6800, RX 7900 XTX, etc.) showed up as
"GPU: Not detected" on the leaderboard despite ROCm working. Corrupts
leaderboard data quality for that population.
Fix: after the existing fallbacks, call `SystemService.getSystemInfo()` and
read `graphics.controllers[0].model`. That path already handles AMD via the
marker file + Ollama log probe added in PR #804, so we're reusing existing
plumbing rather than duplicating detection logic.
## install_nomad.sh changes
The existing AMD detection block already runs lspci. Added a codename parse
step that maps Navi 21/22/23/24, Rembrandt, Phoenix1/Phoenix2, Strix/Strix
Point/Strix Halo, and Navi 31/32/33 to gfx versions, then writes
`/opt/project-nomad/storage/.nomad-amd-gfx`. Unknown codenames write nothing
(admin handles missing-marker case via the backward-compat default).
## Validation
Both bugs were originally surfaced and validated empirically on RX 6800 /
gfx1030 / Ubuntu 24.04 + kernel 6.17 + ollama/ollama:rocm during the #810
filing. Validation grid from that report:
| Run | NOMAD Score | tok/s | GPU detected |
|-----------------------------------------------|-------------|-------|-------------------------|
| Pre-fix (Bug A active) | n/a | 0 | yes, but library=cpu |
| HSA_OVERRIDE removed, Bug B unfixed | 73.8 | 221.6 | "Not detected" |
| Both fixes hot-patched (this PR's behavior) | 73.7 | 216.0 | AMD Radeon RX 6800 |
Local checks: `npm run typecheck` clean, `npm run build` clean.
* feat(content): add custom ZIM library sources with pre-seeded mirrors
Users reported slow download speeds from the default Kiwix CDN. This adds
the ability to browse and download ZIM files from alternative Kiwix mirrors
or self-hosted repositories, all through the GUI.
- Add "Custom Libraries" button next to "Browse the Kiwix Library"
- Source dropdown to switch between Default (Kiwix) and custom libraries
- Browsable directory structure with breadcrumb navigation
- 5 pre-seeded official Kiwix mirrors (US, DE, DK, UK, Global CDN)
- Built-in mirrors protected from deletion
- Downloads use existing pipeline (progress, cancel, Kiwix restart)
- Source selection persists across page loads via localStorage
- Scrollable directory browser (600px max) with sticky header
- SSRF protection on all custom library URLs
Closes#576
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(content): recognize Wikipedia downloads from mirror sources
When Wikipedia is downloaded via a custom mirror instead of the default
Kiwix server, the completion callback now matches by filename instead
of exact URL. This ensures the Wikipedia selector correctly shows
"Installed" status and triggers old-version cleanup regardless of
which mirror was used.
Also handles the case where no Wikipedia selection exists yet (file
downloaded before visiting the selector), creating the record
automatically.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(ZIM): use cheerio for custom mirror directory parsing
* fix(ZIM): use URL constructor for more robust joining
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Jake Turner <jturner@cosmistack.com>
* feat(maps): add regional map downloads via go-pmtiles extract
* address Copilot review feedback on PR #780
- auto-refresh preflight on selection/maxzoom change with 400ms debounce and
requestId stale-safety so the confirm button no longer requires a two-step
"Estimate Size" -> "Start Download" dance
- safeUpdateProgress helper replaces fire-and-forget updateProgress().catch()
pattern so cancelled-job errors (code -1) can't surface as unhandled rejections
- gate world basemap source on worldBasemapReady - when ensureWorldBasemap()
fails we already delete world.pmtiles, so emitting the source was producing
404s on every tile request
- verify go-pmtiles binary SHA256 at image build time; upstream doesn't ship a
checksums file so per-arch hashes are pinned as build args with a regenerate
note when bumping PMTILES_VERSION
Content Updates had three UX problems that compounded:
1. No size column, so users had to guess how big an update would be before
clicking Update All. Upstream /api/v1/resources/check-updates doesn't
return size, so CollectionUpdateService now enriches each update with
a Content-Length HEAD request in parallel (5s timeout, non-fatal on
failure — the row just renders an em-dash).
2. Small ZIM updates (1-8 MB) never appeared in Active Downloads. Two
causes, both fixed: handleApply / handleApplyAll didn't invalidate the
download-jobs query after dispatching, and useDownloads idled at 30s
between polls — enough for a fast job to dispatch, download, and get
cleaned up by removeOnComplete before the next refetch.
3. applyUpdate didn't forward title / totalBytes to RunDownloadJob, so
any update that did briefly surface in Active Downloads had no label
and no byte-count progress, just a filename and a percentage. It now
passes both (matching zim_service's dispatch pattern).
Also parallelized applyAllUpdates so dispatching five updates doesn't
serialize five sequential BullMQ round-trips.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(AI): re-enable AMD GPU acceleration for Ollama via ROCm + HSA override
Re-enables AMD GPU support that was disabled in 77f1868 pending validation
of the ROCm image and device discovery. Validation done 2026-04-28 on a
Minisforum UM890 Pro (Ryzen 9 PRO 8945HS + Radeon 780M iGPU) — Ollama
correctly offloaded all model layers to the iGPU when the container was
started with /dev/kfd + /dev/dri passthrough and HSA_OVERRIDE_GFX_VERSION=11.0.0.
On llama3.2:1b, GPU inference ran at 51.83 tok/s vs 33.16 tok/s on CPU
(same hardware, same prompt) — a 1.56x speedup confirmed by Ollama logs
showing "load_tensors: offloaded 17/17 layers to GPU".
Changes
-------
docker_service.ts
- Restore _discoverAMDDevices() (simplified — pass /dev/dri as a directory
entry, mirroring `docker run --device /dev/dri` behavior, instead of the
prior brittle hardcoded card0/renderD128 fallback that broke on systems
where the AMD GPU enumerates as card1+).
- Restore the AMD branch in _createContainer():
- Switches Ollama image to ollama/ollama:rocm
- Mounts /dev/kfd + /dev/dri via Devices
- Sets HSA_OVERRIDE_GFX_VERSION=11.0.0 (required for unsupported-but-RDNA3
iGPUs like gfx1103; harmless on supported discrete cards)
- KV opt-out via ai.amdGpuAcceleration (default on)
- Mirror the AMD branch in updateContainer():
- Lifted GPU detection above docker.pull() so AMD updates pull :rocm
rather than the standard :targetVersion tag (per-version ROCm tags
aren't always published)
- Replaces stale HSA_OVERRIDE in the inspect-captured env on update,
so containers built before this PR pick up the current value
system_service.ts
- New getOllamaInferenceComputeFromLogs() — parses Ollama startup log line
"msg=\"inference compute\" ... library=CUDA|ROCm ..." which Ollama emits
for both NVIDIA and AMD. Catches silent CPU fallback (e.g. NVML death
after update, or HSA_OVERRIDE failure) that the prior nvidia-smi exec
probe couldn't detect.
- gpuHealth refactored to use log parsing as the primary probe for both
vendors, with nvidia-smi exec retained as the NVIDIA-only secondary
path for hardware enrichment when log parsing has no startup line yet.
- AMD path uses gpu.type KV value (persisted by DockerService._detectGPUType)
+ ai.amdGpuAcceleration opt-out to determine hasRocmRuntime.
types/system.ts
- GpuHealthStatus extended additively: hasRocmRuntime + optional gpuVendor.
types/kv_store.ts
- New ai.amdGpuAcceleration boolean (default-on).
settings/models.tsx, settings/system.tsx
- passthrough_failed banner copy now reads vendor from gpuHealth.gpuVendor
("an AMD GPU" vs "an NVIDIA GPU"). Same Fix button hits the same
force-reinstall endpoint, which now configures AMD correctly.
install_nomad.sh
- AMD detection in verify_gpu_setup() upgraded from a strict-positive
"ROCm not currently available" message to "ROCm acceleration will be
configured automatically." Also tightens the lspci match to display
controller classes (avoids false positives from AMD CPU host bridges,
matching the same fix already in DockerService._detectGPUType).
Auto-remediation
----------------
Issue #755 proposes auto-remediation when gpuHealth.status flips to
passthrough_failed (today the user has to click "Fix: Reinstall AI
Assistant"). When that PR lands, AMD coverage falls out for free since
this PR uses the same passthrough_failed status code via the shared
gpuHealth machinery — #755's guard will need to flip from
hasNvidiaRuntime === true to (hasNvidiaRuntime || hasRocmRuntime).
Closes#124 (AMD GPU support).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(AI): detect AMD GPU presence inside admin container via marker file
The admin container doesn't have lspci installed, and AMD GPUs don't register
a Docker runtime the way NVIDIA does — so DockerService._detectGPUType() and
SystemService.gpuHealth had no way to know an AMD GPU was present.
The previous implementation fell through to lspci, which silently failed inside
the admin container, leaving gpu.type unset and gpuHealth stuck at 'no_gpu'
even on systems with an AMD GPU. (NVIDIA worked because Docker registers the
nvidia runtime, which is reachable via dockerInfo.Runtimes from any container.)
Discovered while testing the AMD acceleration patch on a Minisforum UM890 Pro:
the AMD branch in _createContainer() never fired because _detectGPUType()
returned 'none' even on a host with a working /dev/kfd.
Fix
---
install_nomad.sh writes the host-detected GPU type ('nvidia' | 'amd') to a
marker file in the storage volume the admin container already bind-mounts:
/opt/project-nomad/storage/.nomad-gpu-type → /app/storage/.nomad-gpu-type
DockerService._detectGPUType() reads the marker as a secondary probe (after
the Docker runtime check) — covers AMD detection from inside the container
without requiring lspci or a /dev bind mount.
SystemService falls back to the marker file when KV gpu.type is empty so the
System page reflects AMD presence even before the user installs AI Assistant
for the first time. (Without this, the page would say 'no_gpu' until Ollama
was installed, even on hosts with an AMD GPU detected at install time.)
Verified on NOMAD6 (UM890 Pro, Ubuntu 24.04, 780M iGPU): with the marker file
in place and admin restarted, the patch's AMD branch fires correctly on Force
Reinstall AI Assistant. Resulting nomad_ollama runs ollama/ollama:rocm with
/dev/kfd + /dev/dri passthrough and HSA_OVERRIDE_GFX_VERSION=11.0.0; Ollama
logs show 'library=ROCm compute=gfx1100 ... type=iGPU'. NOMAD's in-product
benchmark on the same hardware climbed from 33.8 tok/s (CPU) to 57.3 tok/s
(GPU) — a 1.69x speedup, with TTFT dropping from 148ms to 66ms.
Migration for existing AMD installs
-----------------------------------
Users on an existing NOMAD install with an AMD GPU have no marker file (the
install script wrote it on a fresh install). Two paths get them on the GPU:
1. Re-run install_nomad.sh — writes the marker, no other side effects
2. Manually: echo amd | sudo tee /opt/project-nomad/storage/.nomad-gpu-type
Either then triggers AMD detection on the next AI Assistant install/reinstall.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(AI): pull ollama/ollama:rocm separately when AMD branch overrides image
The pull-if-missing logic in _createContainer ran against service.container_image
(the DB-pinned tag, e.g. ollama/ollama:0.18.2). The AMD branch then overrode
finalImage to ollama/ollama:rocm — but if that image wasn't already local, the
container creation step failed with "no such image: ollama/ollama:rocm".
Caught while validating on NOMAD2 (Ryzen AI 9 HX 370 + Radeon 890M / RDNA 3.5):
the prior end-to-end test on NOMAD6 had silently passed because the rocm image
was already pulled there from an earlier sidecar test, masking the bug.
Fix: inside the AMD branch, after setting finalImage to ollama/ollama:rocm,
run a parallel _checkImageExists + docker.pull dance for the new tag.
Also confirmed via this validation: the same HSA_OVERRIDE_GFX_VERSION=11.0.0
override works on the 890M (gfx1150 / RDNA 3.5) — Ollama logs report
'library=ROCm compute=gfx1100 description="AMD Radeon 890M Graphics"' and
inference runs at 51.68 tok/s (matching the existing X1 Pro published tile
of 51.7 tok/s on the same hardware class). RDNA 3 (780M, gfx1103) and RDNA
3.5 (890M, gfx1150) both use the same override successfully.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* build(Dockerfile): include pciutils for lspci gpu detection fallback
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Jake Turner <jturner@cosmistack.com>
Adds a check to RAG health to make sure nomad_qdrant is online, if not
then the user will be blocked from clicking any buttons in the KB modal
until they click the start qdrant button and let the container start
There is a new file qdrant_restart_policy_provider.ts, which tries to
ensure that the restart policy always exists for the nomad_qdrant
container even though the policy should have been there when the
container is created.
The VineJS validators in createMarker and updateMarker silently
dropped fields not in their schema. The MapMarker model and DB
include notes and marker_type, and GET responses return them, but
POST and PATCH would not persist them.
updateMarker additionally did not accept latitude/longitude, so
markers could not be repositioned via the API after creation.
- Add notes and marker_type to both validators and model assignments.
- Add latitude/longitude to the update validator.
- Add coordinate range validation on both endpoints.
Closes#768
Some Ollama installs ship nomic-embed-text:v1.5 with the embedding
model's default num_ctx=2048, which the RAG chunker (sized for ~1500
tokens of estimated content with ratio=2 chars/token) can exceed on
dense PDFs. The result is `400 the input length exceeds the context
length` from /api/embed, which then hits the OpenAI-compatible
fallback (which also errors), and surfaces as a BadRequestError.
Pass options.num_ctx=8192 (nomic-embed-text v1.5's RoPE-extrapolated
max) and truncate=true (silent truncation safety net) on every
embed call so we don't depend on the local modelfile defaults.
Reported on #756 by @NC4WD; same root cause as #369 and #670 which
were closed without an actual fix.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(stream): skip compression for Server-Sent Events
The global compression middleware (added in v1.31.0-rc.2) buffers
response writes to determine encoding, which collapses per-token
streaming into a single block delivered after generation completes.
This broke the AI chat streaming UX from v1.31.0-rc.2 onward — text
no longer appears progressively as the model generates it, only at
the end.
Adds a filter to compression() that returns false when the response
Content-Type is text/event-stream. Other responses still go through
the default compression filter (compressible types are still
compressed; e.g. text/html via Brotli).
Reproduced on NOMAD3 v1.31.1: before fix, all SSE chunks for a 1B
model arrive within 10ms of each other after the model finishes.
After fix, tokens arrive at ~150ms intervals as they're generated
on a 12B model, with no Content-Encoding header on the SSE response.
Verified on the same host that /home still returns
Content-Encoding: br for HTML responses.
Closes#781. Reported and bisected by @toasterking
(works in v1.31.0-rc.1, broken from v1.31.0-rc.2 onward).
* fix(stream): use any for filter params to match existing as-any pattern
The compression library types its filter as (req: Request, res: Response)
expecting Express types, but AdonisJS passes raw IncomingMessage/ServerResponse
which is why the surrounding middleware uses `as any` casts at the call site.
The IncomingMessage/ServerResponse types I added are runtime-correct but
fail tsc against the library's declared types.
Drop the typed import in favor of `any` parameters, which matches how the
existing `compress(request.request as any, response.response as any, ...)`
call resolves the same mismatch.
Closes#685
Content Manager now surfaces the on-disk size of each ZIM file alongside
title/summary, and lets users sort the list by Size or Title. Defaults to
Size descending so the largest files are visible first.
- ZimService.list() now stats each file and returns size_bytes
- Content Manager table adds a formatted Size column (via formatBytes)
- Sortable headers for Title and Size with asc/desc toggle
Three bugs in the RAG embedding pipeline, diagnosed and patched by @sbruschke
against v1.31.0 with working before/after chunk counts. All three are
root-cause contributors to #388.
1. scanAndSyncStorage queued every file under /storage/zim/ for embedding,
including Kiwix's generated kiwix-library.xml. EmbedFileJob rejected it
with "Unsupported file type" and the default 30-attempt retry policy
kept it looping on every sync, flooding nomad_admin logs. Now gated on
determineFileType(filePath) !== 'unknown'.
2. hasMoreBatches compared zimChunks.length (section-level chunk count
under the 'structured' strategy) against ZIM_BATCH_SIZE (an article
limit). Because articles emit multiple sections, the two are never
equal for real archives and processing silently stopped after the
first 50 articles. Now gated on articlesInBatch >= ZIM_BATCH_SIZE.
3. extractStructuredContent walked only direct children of <body>, so any
ZIM that wraps content in a container div (Devdocs, Wikipedia,
FreeCodeCamp, React docs, etc.) produced zero sections and silently
embedded zero chunks while reporting success. Now walks the full DOM
via $('body').find('h2, h3, h4, p, ul, ol, dl, table'), with a
whole-body text fallback when the selector walk yields nothing.
Before/after chunk counts confirmed by @sbruschke on v1.31.0:
devdocs_en_git 0 -> 916
devdocs_en_react 0 -> 481
devdocs_en_node 0 -> 423
libretexts_en_eng 1 -> 35 (climbing)
Wikipedia resumed progressing normally through its 6M articles.
Closes#718Closes#719Closes#720Closes#388
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When many ZIMs are already installed locally, a single Kiwix catalog page
(12 items) could return 12 already-installed items, which zim_service
would fully filter out client-side. The endpoint returned items: [] with
has_more: true, and the frontend's infinite-scroll guard
(flatData.length > 0) blocked fetchNextPage — leaving the user with
"No records found" despite plenty of uninstalled ZIMs available.
Backend now accumulates across up to 5 Kiwix fetches (60 items each)
until it has enough post-filter results to return, dedupes by entry id,
advances currentStart by actual entries returned (not requested), and
returns a next_start cursor. The frontend consumes that cursor instead
of computing Kiwix offsets locally, and the flatData.length > 0 guard is
removed so the existing on-mount effect drives bounded auto-fetch when
a short page lands.
The pre-existing has_more off-by-one (compared totalResults against the
input start rather than the post-fetch position) is fixed implicitly.
Diagnosis credit: @johno10661.
Closes#731
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When users set a remote Ollama URL via AI Settings, the local nomad_ollama
container continued running and competed with the remote host for port 11434
and GPU access. Now configureRemote stops the local container on set and
restores it on clear (if still present). Container and its models volume are
preserved so the local install can be re-enabled later.
Closes#662
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Introduces a dedicated page listing third-party ZIM content packs built
by the community. Launches with the two current add-ons (jrsphoto field
manuals, kennethbrewer W3Schools) and explains how to install a ZIM pack
and where to submit a new one for inclusion.
- New doc at admin/docs/community-add-ons.md
- Wired into DocsService DOC_ORDER (slot 4) and TITLE_OVERRIDES so the
hyphen in "Add-Ons" is preserved in the sidebar
- README gets a link under Community & Resources
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a cancel button to in-progress Ollama model downloads and unifies
the Active Model Downloads card layout with the Active Downloads card
used for ZIMs, maps, and pmtiles (byte counts, progress bar, live speed,
status indicator).
Closes#676.
Downloads are now written to `filepath + '.tmp'` and atomically renamed
to the final path only on successful completion. Kiwix globs for `*.zim`
and ZimService filters `.endsWith('.zim')`, so `.tmp` files are invisible
to both during download. The same staging applies to `.pmtiles` map files.
Ref #372
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Corrupted ZIM files cause a native C++ abort (ZimFileFormatError) that
bypasses JS try/catch and kills the process. Add magic number validation
before passing files to @openzim/libzim so invalid files are skipped
gracefully. Also deduplicate Ollama download progress broadcasts — both
within a single stream (skip unchanged percentages) and across concurrent
callers (share one download promise per model).
Co-authored-by: aegisman <aegis@manicode.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add distance scale bar and user-placed location pins to the offline maps viewer.
- Scale bar (bottom-left) shows distance reference that updates with zoom level
- Click anywhere on map to place a named pin with color selection (6 colors)
- Collapsible "Saved Locations" panel lists all pins with fly-to navigation
- Full dark mode support for popups and panel via CSS overrides
- New `map_markers` table with future-proofed columns for routing (marker_type,
route_id, route_order, notes) to avoid a migration when routes are added later
- CRUD endpoints: GET/POST /api/maps/markers, PATCH/DELETE /api/maps/markers/:id
- VineJS validation on create/update
- MapMarker Lucid model
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(downloads): increase retry attempts and backoff for large file downloads
* fix download retry config and abort handling
* use abort reason to detect user-initiated cancels
Henry Estela
Lorenzo Galassi
Jake Turner
Chris Sherwood
johno10661
jakeaturner
akashsalan
Metbcy
chriscrosstalk
Ben Gauger
0xGlitch
John Scherer
Luís Miguel
Aaron Bird