Three bugs in the RAG embedding pipeline, diagnosed and patched by @sbruschke
against v1.31.0 with working before/after chunk counts. All three are
root-cause contributors to #388.
1. scanAndSyncStorage queued every file under /storage/zim/ for embedding,
including Kiwix's generated kiwix-library.xml. EmbedFileJob rejected it
with "Unsupported file type" and the default 30-attempt retry policy
kept it looping on every sync, flooding nomad_admin logs. Now gated on
determineFileType(filePath) !== 'unknown'.
2. hasMoreBatches compared zimChunks.length (section-level chunk count
under the 'structured' strategy) against ZIM_BATCH_SIZE (an article
limit). Because articles emit multiple sections, the two are never
equal for real archives and processing silently stopped after the
first 50 articles. Now gated on articlesInBatch >= ZIM_BATCH_SIZE.
3. extractStructuredContent walked only direct children of <body>, so any
ZIM that wraps content in a container div (Devdocs, Wikipedia,
FreeCodeCamp, React docs, etc.) produced zero sections and silently
embedded zero chunks while reporting success. Now walks the full DOM
via $('body').find('h2, h3, h4, p, ul, ol, dl, table'), with a
whole-body text fallback when the selector walk yields nothing.
Before/after chunk counts confirmed by @sbruschke on v1.31.0:
devdocs_en_git 0 -> 916
devdocs_en_react 0 -> 481
devdocs_en_node 0 -> 423
libretexts_en_eng 1 -> 35 (climbing)
Wikipedia resumed progressing normally through its 6M articles.
Closes#718Closes#719Closes#720Closes#388
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When many ZIMs are already installed locally, a single Kiwix catalog page
(12 items) could return 12 already-installed items, which zim_service
would fully filter out client-side. The endpoint returned items: [] with
has_more: true, and the frontend's infinite-scroll guard
(flatData.length > 0) blocked fetchNextPage — leaving the user with
"No records found" despite plenty of uninstalled ZIMs available.
Backend now accumulates across up to 5 Kiwix fetches (60 items each)
until it has enough post-filter results to return, dedupes by entry id,
advances currentStart by actual entries returned (not requested), and
returns a next_start cursor. The frontend consumes that cursor instead
of computing Kiwix offsets locally, and the flatData.length > 0 guard is
removed so the existing on-mount effect drives bounded auto-fetch when
a short page lands.
The pre-existing has_more off-by-one (compared totalResults against the
input start rather than the post-fetch position) is fixed implicitly.
Diagnosis credit: @johno10661.
Closes#731
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When users set a remote Ollama URL via AI Settings, the local nomad_ollama
container continued running and competed with the remote host for port 11434
and GPU access. Now configureRemote stops the local container on set and
restores it on clear (if still present). Container and its models volume are
preserved so the local install can be re-enabled later.
Closes#662
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Qdrant's upstream default enables anonymous telemetry to telemetry.qdrant.io,
which doesn't match NOMAD's offline-first "zero telemetry" posture. Adding
QDRANT__TELEMETRY_DISABLED=true to the container environment turns it off for
fresh installs and reinstalls.
Existing installs keep their current telemetry-enabled container until the
Qdrant service is force-reinstalled via the Knowledge Base panel or the next
container recreation — Docker bakes Env into containers at create time, so
env changes require a new container.
Closes#742
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Introduces a dedicated page listing third-party ZIM content packs built
by the community. Launches with the two current add-ons (jrsphoto field
manuals, kennethbrewer W3Schools) and explains how to install a ZIM pack
and where to submit a new one for inclusion.
- New doc at admin/docs/community-add-ons.md
- Wired into DocsService DOC_ORDER (slot 4) and TITLE_OVERRIDES so the
hyphen in "Add-Ons" is preserved in the sidebar
- README gets a link under Community & Resources
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a cancel button to in-progress Ollama model downloads and unifies
the Active Model Downloads card layout with the Active Downloads card
used for ZIMs, maps, and pmtiles (byte counts, progress bar, live speed,
status indicator).
Closes#676.
Downloads are now written to `filepath + '.tmp'` and atomically renamed
to the final path only on successful completion. Kiwix globs for `*.zim`
and ZimService filters `.endsWith('.zim')`, so `.tmp` files are invisible
to both during download. The same staging applies to `.pmtiles` map files.
Ref #372
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Corrupted ZIM files cause a native C++ abort (ZimFileFormatError) that
bypasses JS try/catch and kills the process. Add magic number validation
before passing files to @openzim/libzim so invalid files are skipped
gracefully. Also deduplicate Ollama download progress broadcasts — both
within a single stream (skip unchanged percentages) and across concurrent
callers (share one download promise per model).
Co-authored-by: aegisman <aegis@manicode.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When the OpenAI-compatible fallback (/v1/models) is used, models are mapped as { name: m.id, size: 0 } with no details field. Accessing model.details.parameter_size throws `TypeError: Cannot read properties of undefined`, which crashes the React render and causes the entire page to go blank.
Defaults to metric for global audience. Persists choice in localStorage.
Segmented button styled to match MapLibre controls.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add distance scale bar and user-placed location pins to the offline maps viewer.
- Scale bar (bottom-left) shows distance reference that updates with zoom level
- Click anywhere on map to place a named pin with color selection (6 colors)
- Collapsible "Saved Locations" panel lists all pins with fly-to navigation
- Full dark mode support for popups and panel via CSS overrides
- New `map_markers` table with future-proofed columns for routing (marker_type,
route_id, route_order, notes) to avoid a migration when routes are added later
- CRUD endpoints: GET/POST /api/maps/markers, PATCH/DELETE /api/maps/markers/:id
- VineJS validation on create/update
- MapMarker Lucid model
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(downloads): increase retry attempts and backoff for large file downloads
* fix download retry config and abort handling
* use abort reason to detect user-initiated cancels
New defaults:
OLLAMA_NO_CLOUD=1 - "Ollama can run in local only mode by disabling
Ollama’s cloud features. By turning off Ollama’s cloud features, you
will lose the ability to use Ollama’s cloud models and web search."
https://ollama.com/blog/web-searchhttps://docs.ollama.com/faq#how-do-i-disable-ollama%E2%80%99s-cloud-features
example output:
```
ollama run minimax-m2.7:cloud
Error: ollama cloud is disabled: remote model details are unavailable
```
This setting can be safely disabled as you have to click on a link to
login to ollama cloud and theres no real way to do that in nomad outside
of looking at the nomad_ollama logs.
This one can be disabled in settings in case theres a model out there
that doesn't play nice. but that doesnt seem necessary so far.
OLLAMA_FLASH_ATTENTION=1 - "Flash Attention is a feature of most modern
models that can significantly reduce memory usage as the context size
grows. "
Tested with llama3.2:
```
docker logs nomad_ollama --tail 1000 2>&1 |grep --color -i flash_attn
llama_context: flash_attn = enabled
```
And with second_constantine/deepseek-coder-v2 with is based on
https://huggingface.co/lmstudio-community/DeepSeek-Coder-V2-Lite-Instruct-GGUF
which is a model that specifically calls out that you should disable
flash attention, but during testing it seems ollama can do this for you
automatically:
```
docker logs nomad_ollama --tail 1000 2>&1 |grep --color -i flash_attn
llama_context: flash_attn = disabled
```
Surfaces all installed AI models in a dedicated table between Settings
and Active Model Downloads, so users can quickly see what's installed
and delete models without hunting through the expandable model catalog.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(downloads): rich progress, friendly names, cancel, and live status
Redesign the Active Downloads UI with four improvements:
- Rich progress: BullMQ jobs now report downloadedBytes/totalBytes instead
of just a percentage, showing "2.3 GB / 5.1 GB" instead of "78% / 100%"
- Friendly names: dispatch title metadata from curated categories, Content
Explorer library, Wikipedia selector, and map collections
- Cancel button: Redis-based cross-process abort signal lets users cancel
active downloads with file cleanup. Confirmation step prevents accidents.
- Live status indicator: green pulsing dot with transfer speed for active
downloads, orange stall warning after 60s of no data, gray dot for queued
Backward compatible with in-flight jobs that have integer-only progress.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(downloads): fix cancel, dismiss, speed, and retry bugs
- Speed indicator: only set prevBytesRef on first observation to prevent
intermediate re-renders from inflating the calculated speed
- Cancel: throw UnrecoverableError on abort to prevent BullMQ retries
- Dismiss: remove stale BullMQ lock before job.remove() so cancelled
jobs can actually be dismissed
- Retry: add getActiveByUrl() helper that checks job state before
blocking re-download, auto-cleans terminal jobs
- Wikipedia: reset selection status to failed on cancel so the
"downloading" state doesn't persist
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(downloads): improve cancellation logic and surface true BullMQ job states
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Jake Turner <jturner@cosmistack.com>
The assertNotPrivateUrl() function blocked standard loopback and link-local
addresses but could be bypassed using IPv4-mapped IPv6 representations:
- http://[::ffff:127.0.0.1]:8080/ → loopback bypass
- http://[::ffff:169.254.169.254]:8080/ → metadata endpoint bypass
- http://[::]:8080/ → all-interfaces bypass
Node.js normalises these to [::ffff:7f00:1], [::ffff:a9fe:a9fe], and [::]
respectively, none of which matched the existing regex patterns.
Add two patterns to close the gap:
- /^\[::ffff:/i catches all IPv4-mapped IPv6 addresses
- /^\[::\]$/ catches the IPv6 all-zeros address
Legitimate RFC1918 LAN URLs (192.168.x, 10.x, 172.16-31.x) remain allowed.
The App Installation Activity list on the Easy Setup complete page grew
unboundedly, pushing Active Downloads off-screen. Caps the list at ~8
visible items with overflow scrolling, auto-scrolling to keep the latest
activity visible.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Backend returned { error: message } on 400 but frontend expected { message }.
catchInternal swallowed Axios errors and returned undefined, causing a
generic 'An internal error occurred' message instead of the real reason
(already installed, already in progress, not found).
- Fix 400 response shape to { success: false, message } in controller
- Replace catchInternal with direct error handling in installService,
affectService, and forceReinstallService API methods
- Extract error.response.data.message from Axios errors so callers
see the actual server message
The log2 normalization formula `50 * (1 + log2(ratio))` produces negative
values (clamped to 0) whenever the measured value is less than half the
reference. For example, a CPU scoring 1993 events/sec against a 5000
reference gives ratio=0.4, log2(0.4)=-1.32, score=-16 -> 0%.
Fix by dividing log2 by 3 to widen the usable range. This preserves the
50% score at the reference value while allowing below-average hardware
to receive proportional non-zero scores (e.g., 28% for the CPU above).
Also adds debug logging for CPU sysbench output parsing to aid future
diagnosis of parsing issues.
Fixes#415
Model downloads that fail (e.g., when Ollama is too old for a model)
were silently retrying 40 times with no UI feedback. Now errors are
broadcast via SSE and shown in the Active Model Downloads section.
Version mismatch errors use UnrecoverableError to fail immediately
instead of retrying. Stale failed jobs are cleared on retry so users
aren't permanently blocked.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
O Content Manager chamava api.deleteZimFile() para deletar arquivos
ZIM, mas esse método nunca foi implementado na classe API, causando
"TypeError: deleteZimFile is not a function".
O backend (DELETE /api/zim/:filename → ZimController.delete) já
existia e funcionava corretamente — só faltava o método no client
frontend que faz a ponte.
Closes#372
roguesupport.com changed to rogue.support (the actual domain).
Updates href and display text in two places.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
GPU detection results were only applied at container creation time and
never persisted. If live detection failed transiently (Docker daemon
hiccup, runtime temporarily unavailable), Ollama would silently fall
back to CPU-only mode with no way to recover short of force-reinstall.
Now _detectGPUType() persists successful detections to the KV store
(gpu.type = 'nvidia' | 'amd') and uses the saved value as a fallback
when live detection returns nothing. This ensures GPU config survives
across container recreations regardless of transient detection failures.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Failed download jobs persist in BullMQ forever with no way to clear
them, leaving stale error notifications in Content Explorer and Easy
Setup. Adds a dismiss button (X) on failed download cards that removes
the job from the queue via a new DELETE endpoint.
- Backend: DELETE /api/downloads/jobs/:jobId endpoint
- Frontend: X button on failed download cards with immediate refresh
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The security audit report was an internal pre-launch document that
shouldn't be exposed in the user-facing documentation sidebar.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The KV store returns ui.hasVisitedEasySetup as boolean true, but the
comparison checked against string 'true'. Since true !== 'true', the
badge was always shown even after completing Easy Setup.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
After an update completes, the page reloads but the KV store still has
updateAvailable=true from the pre-update check. This causes the banner
to show "Current 1.30.0-rc.1 → New 1.30.0-rc.1" until the user
manually clicks Check Again.
Now triggers a version re-check before the post-update reload so the
KV store is updated and the banner reflects the correct state.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three bugs caused downloads to hang, disappear, or leave stuck spinners:
1. Wikipedia downloads that failed never updated the DB status from 'downloading',
leaving the spinner stuck forever. Now the worker's failed handler marks them as failed.
2. No stall detection on streaming downloads - if data stopped flowing mid-download,
the job hung indefinitely. Added a 5-minute stall timer that triggers retry.
3. Failed jobs were invisible to users since only waiting/active/delayed states were
queried. Now failed jobs appear with error indicators in the download list.
Closes#364, closes#216
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The disk-collector could produce an empty fsSize array when
/host/proc/1/mounts is unreadable, causing the admin UI to fall back
to systeminformation's fsSize which includes tmpfs mounts. This led to
the storage display showing ~1.5 GB (tmpfs /run) instead of the actual
storage capacity.
Two changes:
- disk-collector: fall back to df on /storage when host mount table
yields no real filesystems, since /storage is always bind-mounted
from the host and reflects the actual backing device.
- easy-setup UI: when falling back to systeminformation fsSize, filter
for /dev/ block devices and prefer the largest one instead of blindly
taking the first entry.
Fixes#373
When Ollama isn't installed, every ZIM download dispatches embedding jobs
that fail and retry 30x with 60s backoff. With many ZIM files downloading
in parallel, this exhausts Redis connections with EPIPE/ECONNRESET errors.
Two changes:
1. Don't dispatch embedding jobs when Ollama isn't installed (belt)
2. Use BullMQ UnrecoverableError for "not installed" so jobs fail
immediately without retrying (suspenders)
Closes#351
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add a "Debug Info" link to the footer and settings sidebar that opens a
modal with non-sensitive system information (version, OS, hardware, GPU,
installed services, internet status, update availability). Users can copy
the formatted text and paste it into GitHub issues.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Updated hardware guide price references from $200–$800+ to $150–$1,000+
based on community leaderboard data (41 submissions) and current market
pricing. DDR5 RAM and GPU prices are significantly inflated — budget DDR4
refurbs start at $150, recommended AMD APU builds run $500–$800, and
dedicated GPU builds start at $1,000+. Also noted AMD Ryzen 7 with
Radeon graphics as the community sweet spot in the FAQ.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a new settings page with Ko-fi donation link, Rogue Support
banner, and community contribution options (GitHub, Discord).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add a warm charcoal dark mode ("Night Ops") using CSS variable swapping
under [data-theme="dark"]. All 23 desert palette variables are overridden
with dark-mode counterparts, and ~313 generic Tailwind classes (bg-white,
text-gray-*, border-gray-*) are replaced with semantic tokens.
Infrastructure:
- CSS variable overrides in app.css for both themes
- ThemeProvider + useTheme hook (localStorage + KV store sync)
- ThemeToggle component (moon/sun icons, "Night Ops"/"Day Ops" labels)
- FOUC prevention script in inertia_layout.edge
- Toggle placed in StyledSidebar and Footer for access on every page
Color replacements across 50 files:
- bg-white → bg-surface-primary
- bg-gray-50/100 → bg-surface-secondary
- text-gray-900/800 → text-text-primary
- text-gray-600/500 → text-text-secondary/text-text-muted
- border-gray-200/300 → border-border-subtle/border-border-default
- text-desert-white → text-white (fixes invisible text on colored bg)
- Button hover/active states use dedicated btn-green-hover/active vars
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rotate the HMAC secret used for signing benchmark submissions to the
community leaderboard. The previous secret was compromised (hardcoded
in open-source code and used to submit a fake leaderboard entry).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
WikiHow ZIM files were deprecated by Kiwix after WikiHow requested
removal to protect their content from LLM training harvesting.
Replace with "DIY repair guides and how-to content" which accurately
reflects the iFixit, Stack Exchange, and other how-to content
available in NOMAD's curated collections.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
chriscrosstalk
0xGlitch
Jake Turner
dependabot[bot]
Luís Miguel
Aaron Bird
Ben Gauger
Jake Turner
Henry Estela
cosmistack-bot
David Gross
Sebastion
arn6694
Benjamin Sanders
Tom Boucher
Bortlesboat
Chris Sherwood
LuisMIguelFurlanettoSousa
builder555
Andrew Barnes
orbisai0security