Three changes that together let ARM hardware appear on the leaderboard honestly.
Shipping them separately would leave ARM half-supported either way: without the
image a Pi cannot submit at all, and without the architecture field it submits
but is indistinguishable from x86.
1. PIN THE OFFICIAL MULTI-ARCH SYSBENCH IMAGE
severalnines/sysbench publishes amd64 only, so ARM hosts could not run the
System Benchmark at all — not a graceful failure, the container simply cannot
execute. Apple Silicon could only run it under Rosetta emulation, which distorts
the measurement it is taking, and that is what drove a community macOS fork to
substitute a different benchmark and submit incomparable numbers.
Swaps to ghcr.io/crosstalk-solutions/nomad-sysbench (Debian 12 + sysbench
1.0.20+ds-5, built for linux/amd64 + linux/arm64). One digest covers both
architectures; verified that pulling the pinned manifest-list digest resolves to
arm64 on a Raspberry Pi 5 and amd64 on x86, and that RepoDigests reports the
same manifest-list digest on both — so a single allowlist entry serves both.
No rescoring: 1.0.17 -> 1.0.20 measured 1.25% apart on identical hardware with
identical flags (7170.18 vs 7259.56 events/sec), inside run-to-run noise and
~0.3% on a composite. Both digests are allowlisted server-side, so the fleet can
cross over gradually.
2. REPORT THE DIGEST ACTUALLY RESOLVED
The submission previously sent SYSBENCH_DIGEST, the constant the client was
compiled with. The leaderboard validates that field, but a constant attests to
how a client was BUILT rather than what it RAN, so any build inherits a valid
value simply by carrying the same source.
Now reads it back from the image. Uses RepoDigests (the manifest digest we
pulled by), never Id — Id is the config digest, differs per architecture, and
would never match the allowlist. Falls back to the constant if inspection yields
nothing usable, so a benchmark never fails over provenance metadata.
Still forgeable, and always will be with an open-source client. It moves the bar
from "no effort" to "deliberate", which is the distinction that matters when
judging whether a submission is a mistake or a choice.
3. RECORD CPU ARCHITECTURE AND OS
The leaderboard is a single board across instruction sets by design, with
disclosure as the fairness mechanism. Without an architecture field an ARM result
sits unlabelled beside x86 — exactly what the disclosure exists to prevent.
All three fields come from the Docker daemon, reusing the docker.info() call
_detectRunEnvironment already makes. That is deliberate: inside the admin
container os.arch() and si.osInfo() describe the CONTAINER, not the host being
benchmarked.
cpu_architecture Architecture x86_64 -> amd64, aarch64 -> arm64
os_version OSVersion '24.04' (already structured, no parsing)
os_name OperatingSystem 'Ubuntu 24.04.4 LTS' minus the version
run_environment is kept rather than replaced: "which distro" and "is this
virtualised" are different questions, and WSL2 is a real performance factor.
String handling lives in app/utils/platform_metadata.ts with unit tests, matching
the amd_hsa_override convention, so it is testable without a Docker daemon.
Unknown architectures pass through verbatim rather than being guessed at, and
os_name falls back to the full description whenever the version is missing or
absent from it — an over-long name is harmless, a wrong one is not.
Columns are nullable and the submission fields optional, so results recorded
before this shipped remain submittable.
Closes#1156
Refs #1151
Phase 4 of the NOMAD Score v2 effort (stacked on #1089). Captures the raw
channel values the leaderboard scores from, computes the uncapped v2 score
in-app (byte-matching the leaderboard's server-side recompute), submits the
v2 payload, and surfaces it in the benchmark UI.
- Capture raws: per-channel sysbench values + O_DIRECT disk (W4), single/multi
thread CPU + memory, W6 consistency companions, and run-environment metadata
(run_environment, storage_path_type, gpu_compute_detected, #1016).
- AI hardening: reference model llama3.1:8b, num_predict=256, VRAM eviction
before the run + unload own model after, pre-flight disk check (~6.5GB, only
when the model is uncached), median-of-3 run (W7).
- Score: _calculateNomadScoreV2 (uncapped, reference->1000) byte-matches the
leaderboard recompute. nomad_score_v2 stays null unless a full run with AI.
- Payload: submitToRepository sends the v2 raws + score alongside legacy v1.
- UI: v2 headline score, legacy v1 as secondary, raws/env in details.
- Migration: 13 nullable columns; double (not knex float(8,2)) so millions-scale
raws keep full precision and don't break the leaderboard byte-match.
Reference constant REFERENCE_SCORES_V2.ai_tokens_per_second is still the stale
40.5 placeholder (TODO in source); locking it to the measured 13.2 is a
follow-up gated on the AMD iGPU fix (#1074). Live prod v2 submit is separately
blocked by the leaderboard TTFT floor (tracked in the leaderboard follow-up).
Verified on NOMAD3 (RTX 5060) dev env: migration applies, raws recorded,
v2 score computed, backend + benchmark.tsx typecheck clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Score v2 Phase 1. Safe under v1 — no scoring, weight, reference, or
submission-payload changes; only the benchmark's failure behavior and
forensic metadata.
- Fail-on-parse-miss (W3): the four SCORED sysbench metrics (CPU
events/sec, memory ops/sec, disk read/write MiB/s) now THROW when the
regex misses or the value is <= 0, instead of silently returning 0.
A parse failure (e.g. an upstream image output-format change) now
fails the run with a clear error via the existing _runBenchmark
try/catch, rather than submitting a phantom zero sub-score. Secondary/
informational fields keep their existing defaults.
- Pin sysbench by digest (W3): severalnines/sysbench@sha256:64cd003b...
(was :latest), so a latest-tag format change can't break the parsers
fleet-wide. Digest validated on the NOMAD6 reference build.
- Record provenance (W7): sysbench_digest + ollama_version (from Ollama
/api/version, null-tolerant) stored on each result. New nullable
columns + additive migration.
Verified: pinned digest pulls/runs/parses on NOMAD3; migration applies
(columns present); typecheck clean.
Part of the NOMAD Score v2 effort.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(benchmark): Require full benchmark with AI for community sharing
Only allow users to share benchmark results with the community leaderboard
when they have completed a full benchmark that includes AI performance data.
Frontend changes:
- Add AI Assistant installation check via service API query
- Show pre-flight warning when clicking Full Benchmark without AI installed
- Disable AI Only button when AI Assistant not installed
- Show "Partial Benchmark" info alert for non-shareable results
- Only display "Share with Community" for full benchmarks with AI data
- Add note about AI installation requirement with link to Apps page
Backend changes:
- Validate benchmark_type is 'full' before allowing submission
- Require ai_tokens_per_second > 0 for community submission
- Return clear error messages explaining requirements
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(benchmark): UI improvements and GPU detection fix
- Fix GPU detection to properly identify AMD discrete GPUs
- Fix gauge colors (high scores now green, low scores red)
- Fix gauge centering (SVG size matches container)
- Add info tooltips for Tokens/sec and Time to First Token
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix(benchmark): Extract iGPU from AMD APU CPU name as fallback
When systeminformation doesn't detect graphics controllers (common on
headless Linux), extract the integrated GPU name from AMD APU CPU model
strings like "AMD Ryzen AI 9 HX 370 w/ Radeon 890M".
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat(benchmark): Add Builder Tag system for community leaderboard
- Add builder_tag column to benchmark_results table
- Create BuilderTagSelector component with word dropdowns + randomize
- Add 50 adjectives and 50 nouns for NOMAD-themed tags (e.g., Tactical-Llama-1234)
- Add anonymous sharing option checkbox
- Add builder tag display in Benchmark Details section
- Add Benchmark History section showing all past benchmarks
- Update submission API to accept anonymous flag
- Add /api/benchmark/builder-tag endpoint to update tags
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat(benchmark): Add HMAC signing for leaderboard submissions
Sign benchmark submissions with HMAC-SHA256 to prevent casual API abuse.
Includes X-NOMAD-Timestamp and X-NOMAD-Signature headers.
Note: Since NOMAD is open source, a determined attacker could extract
the secret. This provides protection against casual abuse only.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Add comprehensive benchmarking capability to measure server performance:
Backend:
- BenchmarkService with CPU, memory, disk, and AI benchmarks using sysbench
- Database migrations for benchmark_results and benchmark_settings tables
- REST API endpoints for running benchmarks and retrieving results
- CLI commands: benchmark:run, benchmark:results, benchmark:submit
- BullMQ job for async benchmark execution with SSE progress updates
- Synchronous mode option (?sync=true) for simpler local dev setup
Frontend:
- Benchmark settings page with circular gauges for scores
- NOMAD Score display with weighted composite calculation
- System Performance section (CPU, Memory, Disk Read/Write)
- AI Performance section (tokens/sec, time to first token)
- Hardware Information display
- Expandable Benchmark Details section
- Progress simulation during sync benchmark execution
Easy Setup Integration:
- Added System Benchmark to Additional Tools section
- Built-in capability pattern for non-Docker features
- Click-to-navigate behavior for built-in tools
Fixes:
- Docker log multiplexing issue (Tty: true) for proper output parsing
- Consolidated disk benchmarks into single container execution
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
chriscrosstalk
Chris Sherwood