Rebuilt on top of dev's RFC #883 state-machine UI rather than the now-defunct
StoredFile shape:
- Extend StoredFileInfo with fileName/size/uploadedAt/isUserUpload
- Populate metadata from on-disk stats in RagService.getStoredFiles
- Add fileSourceSchema validator + getFileContent/downloadFile endpoints
scoped to the uploads directory only (tighter than the original PR — matches
docs_service traversal pattern)
- KnowledgeBaseModal: sortable Size and Uploaded columns; View/Download
buttons on upload-bucket rows; new FileViewerModal for in-browser text
preview. Bucket grouping preserved — sort applies within each bucket.
- Use formatBytes from ~/lib/util rather than redefining
The Stored Files "partial stall" warning compares chunks in Qdrant against an
expected count from the ratio registry. The registry has an empty-pattern
catch-all (100 chunks/MB) that matches any filename, so a ZIM that matches no
specific pattern still gets a size-based estimate. For archives that are mostly
PDFs, images, or link-out stubs (e.g. irp.fas.org military-medicine), byte size
wildly over-predicts embeddable text: a 75 MB ZIM estimates ~7,236 chunks but
produces ~1, tripping a false "ingestion may have stalled" warning that re-embed
can't clear.
The catch-all is fine for rough aggregate disk-cost estimates, but it should not
drive a per-file stall signal. Add an `ignoreCatchAll` option to the ratio
lookup that excludes the empty-pattern row (returning null when only the
fallback would match), and use it in the warnings path so partial_stall only
fires when the registry has a *specific* expectation for the file. Files that
match a real pattern (wikipedia_, devdocs_, ifixit_, ...) are unaffected;
disk-cost/batch estimates keep using the fallback.
Closes#913
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Users often saw the assistant disclaim ("Sorry, I couldn't find specific
context regarding X, but here's a general answer...") even when material
that directly answered the query was embedded. Two compounding causes,
both on the read/generation side:
1. The rag_context system prompt explicitly authorized the hedge: it told
the model to fall back to general knowledge and "acknowledge the
limitations," and mandated "According to the information available..."
citations that pushed small models into meta-commentary preambles.
Rewrote it to treat the retrieved context as the primary, authoritative
source, lead with the answer, fall back to general knowledge *silently*,
and never emit "couldn't find specific context" phrasing, while still allowing
the model itself to make some "sanity-check" decisions if the retrieved context
seems wildly incorrect or unrelated to the user's query.
2. The injected context labeled each block with a raw relevance score
(e.g. "Relevance: 42.3%"). nomic-embed cosine scores for genuinely
relevant passages sit ~0.4-0.6, so the number primed the model to
distrust correct context. Dropped the model-visible score (it stays in
the logs) and replaced it with a neutral source-title label.
Also added a conservative, score-scaled heading boost in rerankResults:
when query keywords match a chunk's section/article title (ZIM metadata we
already fetch), nudge its rank. Same diminishing-returns shape as the
existing boosts and gated behind the existing semantic-quality threshold,
so it can't promote a weak match.
Scope note: this removes the visible hedge and improves ranking of
already-retrieved chunks. It does NOT fix retrieval recall — diluted
1500-token chunks that never reach dense search's top-k are unaffected.
That's a follow-up (smaller chunks + BM25/RRF hybrid).
The Stored Files modal and per-file warnings panel both enumerate
distinct 'source' values in the qdrant content collection. The
existing implementation scrolls every point in the collection, 100 per
page, just to learn the unique sources. On a fully-ingested NOMAD this
is brutally slow: NOMAD3's 3.1M-point collection takes 50+ seconds to
return its ~40 distinct sources, and the same scan runs twice per modal
open (getStoredFiles + computeFileWarnings each pay it independently).
Qdrant 1.10+ supports a facet aggregation that returns the distinct
values of a payload field with their counts in a single call.
@qdrant/js-client-rest@1.16.2 (already pinned) exposes this as
`client.facet()`. Swap the three scroll loops:
- RagService.getStoredFiles
- RagService.computeFileWarnings
- RagService.\_scanAndSync (the source-set used for skip-already-embedded
decisions)
As a bonus, computeFileWarnings's per-source chunk count comes back
inline from the facet, so the secondary scroll that was incrementing a
counter is also gone.
`exact: true` everywhere because the counts feed Warning A's
zero_chunks decision and could mis-fire near thresholds otherwise. New
RagService.FACET_SOURCE_LIMIT = 10000 caps the facet response; real
NOMADs run ~100 sources max, this is comfortable headroom.
Expected impact on NOMAD3 (3M points, 40 sources): ~50s -> ~100ms per
endpoint. On an empty/fresh KB: no measurable change either way.
Lift the hardcoded 'nomic-embed-text:v1.5' string out of both
RagService and OllamaService into a shared EMBEDDING_MODEL_NAME
constant in constants/ollama.ts. The duplicate in OllamaService
existed only to dodge a circular import with RagService; the
constants module has no service imports, so a shared constant
eliminates both the duplication and the drift risk called out
in the inline "keep in sync" comment.
Return a discriminated `EmbedSingleFileResult` from `RagService.embedSingleFile`
with `code: 'not_found' | 'inflight' | 'delete_failed' | 'dispatch_failed'` on
failure. `RagController.embedFile` now maps those codes to the correct status
instead of collapsing every failure to 409:
- not_found → 404
- inflight → 409
- delete_failed → 500
- dispatch_failed → 500
The `code` is also included in the JSON body so clients can branch without
string-matching `error`.
Closes the Manual-mode UX dead-end: after toggling 'Auto-index new content
for AI?' to Manual, a freshly-downloaded ZIM (or any pending_decision file)
had no UI path to opt in for embedding short of the global Sync Storage /
Re-embed All bulk actions. Per RFC #883 §5, each Stored Files row now
carries a state pill and an adaptive single-button action.
State pill (left of any existing warning chips):
- 'Indexed' — green; row had chunks in Qdrant or state row is 'indexed'
- 'Not Indexed' — neutral; state is pending_decision or browse_only
- 'Failed' — red
- 'Stalled' — amber
- admin_docs collapsed row has no pill ('Managed by NOMAD' carries it)
Adaptive action button (paired with the existing Delete button per row):
- pending_decision → 'Index' (force=false)
- browse_only → 'Index' (force=true)
- failed / stalled → 'Retry' (force=true)
- indexed + warning chip → 'Re-embed' (force=true; confirm modal first)
- indexed healthy / null → no action button (bulk Re-embed All covers it)
Backend: GET /api/rag/files now returns
{ files: Array<{ source, state, chunksEmbedded }> }
instead of a flat string[]. State + chunk-count come from a single
KbIngestState query unioned into the existing Qdrant-derived source list
(no new round trips). New POST /api/rag/files/embed validates the source is
known, refuses if any inflight job already targets the same filePath
(prevents double-click duplicate-chunk hazard), pre-deletes Qdrant points
when force=true, then dispatches via the existing _dispatchEmbedJobsFor
helper used by reembedAll.
Per-file Re-embed (force=true on an already-indexed file) routes through a
StyledModal confirmation since it deletes existing vectors before queueing
a fresh job — same destructive-action weight as Delete's inline confirm but
heavier since it affects search until the rebuild finishes.
Folds in PR #907's blank-screen fix because my new render needs the same
generic restored: `<StyledTable<KbFileGroup>>` and `record.displayName`
(instead of the unresolved `sourceToDisplayName(record.source)` that ships
in rc.5 and ReferenceErrors on modal open). PR #907 also adds title
tooltips on the three bulk-action buttons; those tooltips are NOT included
here — let PR #907 land first or independently for that part.
Multi-select bulk-opt-in deferred per discussion: most Manual-mode users
ingest 1-2 files at a time, the existing global toggle covers the bulk
case, and checkboxes would expand scope past what rc.6 should hold. Will
file a follow-up issue for an 'Index N pending files' single-click button
once this lands.
Tests-in-PR scope was limited to keeping `kb_file_grouping.spec.ts` green
after the StoredFileInfo[] signature change (added asInfos() wrapper).
Dedicated unit tests for embedSingleFile (unknown source / inflight refused
/ force=true delete-then-dispatch) and the new state-pill rendering will
land in a follow-up PR alongside Playwright coverage of the row actions.
Verification path: NOMAD3 currently runs project-nomad-admin:integration-
rc6-preview (PRs #907 + #908 atop rc.5). After this branch is built into a
new integration tag, I'll re-run targeted Playwright UAT on the KB modal
covering: state pill rendering per state, Index click on pending_decision
opts in cleanly, Retry on failed re-dispatches successfully, Re-embed
confirmation modal copy + delete-then-dispatch on the military-medicine
partial-stall row, and Delete flow untouched.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When a user opens AI Chat with content available but no global ingest
policy yet recorded, surface a one-time banner above the chat header
asking how they want new content handled:
- 'Index existing content' -> sets rag.defaultIngestPolicy=Always and
triggers a sync so pending_decision files queue immediately
- 'Maybe later' -> sets policy=Manual; existing and future content
waits in pending_decision until the user opts in from the KB modal
After either button is clicked the banner never reappears, because both
write the policy KV (the same one #894 manages via the KB modal toggle).
There is intentionally no 'dismiss without deciding' X — that would just
re-show the banner forever.
Backend
- New GET /api/rag/policy-prompt-state returns
{shouldPrompt, hasContent, totalFiles}
- RagService.getPolicyPromptState() reads KVStore('rag.defaultIngestPolicy')
and counts kb_ingest_state rows; shouldPrompt is true only when policy
is null AND scanner has seen >=1 file (avoids prompting on empty NOMADs)
Frontend
- New KbPolicyPromptBanner component (~120 LOC) handles the two-button
decision flow with optimistic loading state, success/error toasts, and
invalidates kbPolicyPromptState + ingestPolicy + embed-jobs + storedFiles
on success
- Mounted in components/chat/index.tsx as the first child of the main
content column so it sits above the chat title bar without taking space
when shouldPrompt is false (renders nothing)
- Reads aiAssistantName from Inertia page props so banner copy matches
the user's chosen assistant name
Stacks on feat/kb-policy-toggle (#894) because the policy KV mechanism
it writes through is introduced there. Both can land in rc.5; this PR
auto-rebases to rc once #894 merges.
Existing users on first upgrade to v1.32.0 will see this banner on first
chat visit post-upgrade — an explicit opt-in moment for content that was
already on disk. New users see it the first time they have curated
content downloaded.
Closes the 'zero_chunks warning has no row to attach to' gap surfaced by
the 2026-05-14 integration UAT. Before this fix RagService.getStoredFiles
returned only file paths that appeared in Qdrant's payload.source — so
files with 0 embedded chunks (video-only ZIMs, browse_only opt-outs,
ingestions that failed before producing any chunks) silently disappeared
from the KB panel's Stored Files table.
The fix unions the Qdrant scroll result with the disk-backed file paths
recorded in kb_ingest_state. Effect:
- lrnselfreliance_en_all_2025-12.zim (3.97 GB video-only ZIM, 0 chunks)
now appears in the table, picks up its zero_chunks warning chip
- Files in pending_decision under Manual policy show up so the user can
see what's waiting for opt-in
- Files in browse_only / failed states have a row for future per-card
Retry / Re-index actions (forthcoming, blocked on #886)
The state-machine query is wrapped in its own try/catch so a transient
DB error degrades to the Qdrant-only list rather than 500-ing the whole
panel — same defensive posture as the outer try/catch.
Stacks on feat/kb-ingest-state-machine (#888) because the union depends
on the kb_ingest_state table that PR introduces. Will rebase to rc once
#888 merges. Completes the second half of #895's warning surface; the
first half (partial_stall) already worked because those files have at
least some chunks in Qdrant.
`computeFileWarnings()` previously caught all errors and returned an empty
map, which the frontend rendered as "every file is healthy" — reintroducing
exactly the silent-failure mode this surface exists to expose.
Return `{ ok, warnings }`; flip `ok: false` from the catch. KB modal renders
an inline amber notice under the Stored Files header when `ok === false`,
leaving per-row warning rendering untouched. Transient failures self-heal on
the next 30s poll; no toast spam.
Surfaces two silent failure modes that the prior binary
"any-chunks-in-Qdrant ⇒ embedded" check could not distinguish from
healthy ingestion:
- **Warning A — Zero-chunk file** (file_size > 100 MB, chunks = 0)
Fires on video-only / image-only ZIMs (`lrnselfreliance_en_all`,
TED talks, etc.) that the pipeline completes "successfully" with no
extractable text. AI Assistant literally cannot reference these.
- **Warning B — Partial-embed stall** (chunks < 50% of expected from
the ratio registry). Surfaces the simple_wiki "266 of 600,000 chunks"
case observed during NOMAD1 ingestion testing — previously these
looked identical to fully-completed embeds in the UI.
Both warnings render only when their condition is met (silent by
default; noisy only on real problems).
Base is `feat/kb-ratio-registry` (#891) because Warning B's "expected
chunks" estimate comes from `KbRatioRegistry.estimateChunks()`. GitHub
fast-forwards to `rc` once #891 merges.
- `app/utils/kb_warning_decision.ts` — pure `decideWarnings(inputs)`
with thresholds (`100 MB`, `0.5×`) as exported constants. 10 unit
tests cover the healthy case, both warnings, the under/at/over
boundary, the registry-miss suppression, and the video-only registry
case (`expectedChunks: 0` correctly skips Warning B).
- `RagService.computeFileWarnings()` — single Qdrant scroll tallies
chunks per source, filesystem walk fills in zero-chunk files,
ratio registry estimates the expectation, decision function emits.
- New endpoint `GET /api/rag/file-warnings` returns
`Record<source, FileWarning[]>` (sources with no warnings are
omitted, so the frontend can `warnings[source] ?? []` for clean
defaults).
- KB modal: warnings render inline under the file name as amber-tinted
pills. Polled every 30s alongside the existing health check.
- Warning C — chunks skipped due to length. PR #890 (#881 fix) prevents
the silent drop at the embed boundary, so the underlying condition
shouldn't fire anymore. If we still want to surface "we truncated
N chunks to fit", that needs separate `skipped_count` tracking in
EmbedFileJob — a Phase 2 follow-up.
- Suppressing Warning B during active mid-ingestion. The user can cross-
reference the Processing Queue to know it's in-flight; suppressing
warnings while a job runs would mask real stalls where the job died
mid-batch. Will revisit when per-card status is wired through.
- Use of `kb_ingest_state.chunks_embedded` (#888) as the chunk count
source. This PR uses Qdrant scroll directly so it can land
independently of #888.
- 10 new unit tests on `decideWarnings`, all pass
- Type-check clean
- Hot-patch + browser smoke test deferred until #891 lands (the ratio
registry needs to exist in the DB for `estimateChunks()` to return
non-null estimates — without it, only Warning A fires which is still
useful but Warning B stays dormant)
* feat(KB): per-file ingest state machine (Phase 1 of RFC #883)
Adds a persistent state machine for AI knowledge-base ingestion so the
scanner can distinguish "fully indexed", "user opted out", "failed", and
"stalled" from each other — none of which were derivable from the prior
binary "any chunks in Qdrant ⇒ embedded" check.
## What lands
- New table `kb_ingest_state` keyed by `file_path` with enum state column
(`pending_decision | indexed | browse_only | failed | stalled`).
Independent of `installed_resources` so it covers both curated downloads
and manually-uploaded KB files.
- New KV key `rag.defaultIngestPolicy` (string: `Always | Manual`).
Registered now but not consumed yet — JIT prompt + wizard step land in
Phase 3 of the RFC.
- `EmbedFileJob.handle` writes state on terminal outcomes:
- Success (final batch) → `indexed` + chunks count
- `UnrecoverableError` → `failed` + error message
- Retryable errors are left to BullMQ's existing retry path
- `scanAndSyncStorage` swaps the binary qdrant check for a state-aware
decision tree (see `decideScanAction`). Existing installs auto-backfill
on first scan: files with chunks in Qdrant but no state row become
`indexed`; new files start as `pending_decision`.
- `deleteFileBySource` drops the state row last, so removed files
disappear entirely instead of leaving an orphan that the next scan
would re-dispatch into nothing.
## What does NOT land here
- Ratio registry (separate PR) — needed for partial-stall detection and
cost estimates, but a separable concern.
- #880 follow-up initial-progress anchor (separate tiny PR).
- Phase 2 UI (status pill, per-card actions, conditional warnings).
- Phase 3 policy surfaces (wizard step, JIT prompt, guardrail modal).
- PR #886's bulk-action hookup — `_deletePointsBySource` / Re-embed All
/ Reset & Rebuild would also want to set state, but #886 isn't merged
yet; that wiring goes in a follow-up once #886 lands.
## Target
This is forward work for v1.40.0 (RFC #883). Branching off `rc` because
that's the current latest base and post-GA Jake will sync rc→dev; a
retarget at PR-open time is a fast-forward if requested.
## Tests
- 9 new unit tests for `decideScanAction` covering all five states plus
the no-row / chunks-present / chunks-missing combinations
- Type-check clean
- Smoke-tested end-to-end on NOMAD3 via hot-patch:
- Backfill: 5 ZIMs + 2 KB uploads with existing chunks in Qdrant all
came back `indexed` on first scan
- Pending dispatch: a video-only ZIM with no chunks (`lrnselfreliance`)
came back `pending_decision` and was correctly re-dispatched (Bull
deduped to its historical `:completed` jobId — bgauger's #886 fix
drains that)
- Delete hook: deleting a KB upload via `DELETE /api/rag/files`
removed both the disk file and the state row
* feat(KB): Always/Manual ingest policy toggle (RFC #883 §1/§4)
Activates the `rag.defaultIngestPolicy` KV registered in Phase 1
(#888) so users on a fresh install (or anyone who picks Manual mode)
no longer get every new ZIM auto-dispatched to the embed pipeline.
## Stacks on #888
This PR's base is `feat/kb-ingest-state-machine` (#888). The state
machine has to be in place for the decision function to be policy-aware;
GitHub will fast-forward the base to `rc` once #888 merges.
## Backend changes
- `decideScanAction` now takes a `policy: 'Always' | 'Manual'` argument
(defaults to `Always` for backward compatibility).
- New `ScanAction` kind: `create_pending`. Manual mode records that the
scanner has seen a new file (so the UI can surface a per-card Index
affordance later) without dispatching an EmbedFileJob.
- `scanAndSyncStorage` reads the KV and passes it through. The scan-result
log line now includes the active policy and a `waiting on user` count
for Manual-mode hits.
- `rag.defaultIngestPolicy` added to `SETTINGS_KEYS` so it's reachable
through the existing `GET/PATCH /api/system/settings` surface — no new
endpoint.
## Frontend changes
- New section in the KB panel between "Why upload" and "Processing Queue":
"Auto-index new content for AI? [Always | Manual]" — segmented radio
with copy explaining the 5-10× disk multiplier. Default Always.
- `useQuery('ingestPolicy')` reads the current value; clicking the
inactive option mutates and shows a notification confirming the new
behavior.
## Tests
- 14 unit tests on `decideScanAction` (was 9) — split into Always-mode
cases (preserves Phase 1's contract) and Manual-mode cases
(`create_pending`, `pending_decision → skip`, etc.).
- Type-check clean.
- Hot-patch + browser verification deferred until #888 lands; the state
machine smoke-tested cleanly on NOMAD3 in #888's PR, and this PR's
decision-tree changes are exhaustively unit-tested.
## RFC open question §3 — policy-change re-trigger
Switching Manual → Always doesn't auto-dispatch existing `pending_decision`
rows immediately. The next scan re-evaluates and dispatches them under
the new policy. This matches the RFC's "treat the switch as I've-
thought-about-it" instinct for the guardrail; full guardrail
implementation lands in Phase 3 task 14.
---------
Co-authored-by: Jake Turner <52841588+jakeaturner@users.noreply.github.com>
Adds a persistent state machine for AI knowledge-base ingestion so the
scanner can distinguish "fully indexed", "user opted out", "failed", and
"stalled" from each other — none of which were derivable from the prior
binary "any chunks in Qdrant ⇒ embedded" check.
## What lands
- New table `kb_ingest_state` keyed by `file_path` with enum state column
(`pending_decision | indexed | browse_only | failed | stalled`).
Independent of `installed_resources` so it covers both curated downloads
and manually-uploaded KB files.
- New KV key `rag.defaultIngestPolicy` (string: `Always | Manual`).
Registered now but not consumed yet — JIT prompt + wizard step land in
Phase 3 of the RFC.
- `EmbedFileJob.handle` writes state on terminal outcomes:
- Success (final batch) → `indexed` + chunks count
- `UnrecoverableError` → `failed` + error message
- Retryable errors are left to BullMQ's existing retry path
- `scanAndSyncStorage` swaps the binary qdrant check for a state-aware
decision tree (see `decideScanAction`). Existing installs auto-backfill
on first scan: files with chunks in Qdrant but no state row become
`indexed`; new files start as `pending_decision`.
- `deleteFileBySource` drops the state row last, so removed files
disappear entirely instead of leaving an orphan that the next scan
would re-dispatch into nothing.
## What does NOT land here
- Ratio registry (separate PR) — needed for partial-stall detection and
cost estimates, but a separable concern.
- #880 follow-up initial-progress anchor (separate tiny PR).
- Phase 2 UI (status pill, per-card actions, conditional warnings).
- Phase 3 policy surfaces (wizard step, JIT prompt, guardrail modal).
- PR #886's bulk-action hookup — `_deletePointsBySource` / Re-embed All
/ Reset & Rebuild would also want to set state, but #886 isn't merged
yet; that wiring goes in a follow-up once #886 lands.
## Target
This is forward work for v1.40.0 (RFC #883). Branching off `rc` because
that's the current latest base and post-GA Jake will sync rc→dev; a
retarget at PR-open time is a fast-forward if requested.
## Tests
- 9 new unit tests for `decideScanAction` covering all five states plus
the no-row / chunks-present / chunks-missing combinations
- Type-check clean
- Smoke-tested end-to-end on NOMAD3 via hot-patch:
- Backfill: 5 ZIMs + 2 KB uploads with existing chunks in Qdrant all
came back `indexed` on first scan
- Pending dispatch: a video-only ZIM with no chunks (`lrnselfreliance`)
came back `pending_decision` and was correctly re-dispatched (Bull
deduped to its historical `:completed` jobId — bgauger's #886 fix
drains that)
- Delete hook: deleting a KB upload via `DELETE /api/rag/files`
removed both the disk file and the state row
Co-authored-by: Jake Turner <52841588+jakeaturner@users.noreply.github.com>
Before this change, the Active Downloads / Processing Queue UI showed the
ingestion progress gauge jumping wildly during multi-batch ZIM ingestion
(e.g. 5% → 88% → 27% → 5% → 56% → 36% over ~60 seconds for cooking SE).
Each continuation batch is a separate BullMQ job, and `EmbedFileJob.handle()`
reported `job.progress` in two different reference frames depending on
where it was in the batch lifecycle:
- During-batch (via the onProgress callback): 5% → 95% scaled across
"% through this batch's chunks"
- End-of-batch (just before dispatching the next): overwritten to
`(nextOffset / totalArticles) * 100` — % through the whole file
- Next continuation batch starts with progress = 5% explicitly, then
climbs through the per-batch range again
`listActiveJobs()` returns the latest active BullMQ job's progress. With
GPU-accelerated ingestion completing a batch every ~4 seconds, the UI
saw the jobId rotate constantly and the gauge whipsaw between the two
reference frames.
`totalArticles` was already wired through the EmbedFileJob params shape
and used end-of-batch — but RagService never actually populated it,
so any frame-scaling that depended on it silently fell back to the
per-batch range. Two fixes together:
1. `ZIMExtractionService.extractZIMContent()` now returns
`{ chunks: ZIMContentChunk[]; totalArticles: number }` instead of a
raw chunks array, surfacing `archive.articleCount` to the caller.
Single caller (rag_service) updated to destructure.
2. `RagService.processZimFile()` includes `totalArticles` in its result
so `EmbedFileJob.dispatch()` can propagate it to the continuation
batch (which the existing code already does via
`totalArticles: totalArticles || result.totalArticles`).
3. `EmbedFileJob`'s onProgress callback scales the service-reported
per-batch percent into the overall-file frame when `totalArticles`
is known: `((batchOffset + (percent/100) * ZIM_BATCH_SIZE) /
totalArticles) * 100`. Capped at 99% to leave room for the explicit
100% set at file completion. Falls back to the original 5-95% range
for single-batch files (uploaded PDFs/txts) where totalArticles is
undefined — the gauge then represents % through the only batch,
which is what the UI expects for one-shot files.
Validated on NOMAD8 (RX 6800, ROCm-accelerated nomic):
- devdocs python (small, ~1500 articles): batch progressions seen
monotonically across continuation jobIds:
1501@30% → 1510@33% → 1514@43% → 1518@52%.
- ifixit (huge, ~100k articles): stays near 3% for the first many
batches at offset 0..3000 — correct, the file is enormous.
- wikipedia_en_medicine (large, ~70k articles): stays near 0-1% for
the first batches — also correct.
- Brief 0-5% blip on continuation handoff (the explicit
`safeUpdateProgress(job, 5)` at batch start, before the first
onProgress callback fires) — visible but quickly resolves to the
overall-frame value. No more 5% ↔ 88% chaos.
Adds a check to RAG health to make sure nomad_qdrant is online, if not
then the user will be blocked from clicking any buttons in the KB modal
until they click the start qdrant button and let the container start
There is a new file qdrant_restart_policy_provider.ts, which tries to
ensure that the restart policy always exists for the nomad_qdrant
container even though the policy should have been there when the
container is created.
Three bugs in the RAG embedding pipeline, diagnosed and patched by @sbruschke
against v1.31.0 with working before/after chunk counts. All three are
root-cause contributors to #388.
1. scanAndSyncStorage queued every file under /storage/zim/ for embedding,
including Kiwix's generated kiwix-library.xml. EmbedFileJob rejected it
with "Unsupported file type" and the default 30-attempt retry policy
kept it looping on every sync, flooding nomad_admin logs. Now gated on
determineFileType(filePath) !== 'unknown'.
2. hasMoreBatches compared zimChunks.length (section-level chunk count
under the 'structured' strategy) against ZIM_BATCH_SIZE (an article
limit). Because articles emit multiple sections, the two are never
equal for real archives and processing silently stopped after the
first 50 articles. Now gated on articlesInBatch >= ZIM_BATCH_SIZE.
3. extractStructuredContent walked only direct children of <body>, so any
ZIM that wraps content in a container div (Devdocs, Wikipedia,
FreeCodeCamp, React docs, etc.) produced zero sections and silently
embedded zero chunks while reporting success. Now walks the full DOM
via $('body').find('h2, h3, h4, p, ul, ol, dl, table'), with a
whole-body text fallback when the selector walk yields nothing.
Before/after chunk counts confirmed by @sbruschke on v1.31.0:
devdocs_en_git 0 -> 916
devdocs_en_react 0 -> 481
devdocs_en_node 0 -> 423
libretexts_en_eng 1 -> 35 (climbing)
Wikipedia resumed progressing normally through its 6M articles.
Closes#718Closes#719Closes#720Closes#388
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Benjamin Smith
jakeaturner
Chris Sherwood
chriscrosstalk
Jake Turner
Henry Estela
arn6694