RomanNum3ral
/
project-nomad
mirror of https://github.com/Crosstalk-Solutions/project-nomad.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

feat(benchmark): lock Score v2 AI reference to 13.2 (measured, was placeholder) 

Replaces the stale 40.5 placeholder (a llama3.2:1b CPU-era number) with 13.2,
measured on the Reference Build (NOMAD6, 780M) under the v2 AI harness with
llama3.1:8b and iGPU acceleration (OLLAMA_IGPU_ENABLE — the provisioning fix in
PR #1074). This is the AI channel's reference for the uncapped v2 score.

Must ship together with #1074 in v1.34.0: the reference assumes iGPU-accelerated
AMD boxes, so shipping it without the iGPU fix would score AMD installs on CPU
numbers against a GPU bar. The exact value gets a final confirm against the
shipping Ollama config before GA.

Must stay byte-identical to the leaderboard's score_service.ts REFERENCE_SCORES_V2
(the server recomputes the score on submit); the matching leaderboard change is a
sibling PR. Changing the leaderboard reference has no effect on existing rows —
the v1->v2 backfill neutralizes the AI channel (ratio 1 regardless of the
reference), and there are no real v2 submissions yet.

Sibling to the Score v2 app client (#1094); stacked on it (same file/constant).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris Sherwood 2026-07-13 12:27:39 -07:00
parent 651df80dc8
commit 3632461380
1 changed files with 6 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
admin/app/services/benchmark_service.ts
View File

@ -69,11 +69,12 @@ const BENCHMARK_VERSION = '2.0.0'
// with direct-I/O disk 2026-07-12. There are NO clamps here; outlier control is
// the server's rejection gates + quarantine, not the score math.
const REFERENCE_SCORES_V2 = {
// TODO(score-v2): STALE — 40.5 was measured on llama3.2:1b. The reference model
// is now llama3.1:8b; re-measure this on the Reference Build (NOMAD6) with the
// v2 harness (unload + num_predict + median-of-3) and update here AND in the
// leaderboard's score_service.ts (must byte-match) before v2 ships.
ai_tokens_per_second: 40.5,
// Measured on the Reference Build (NOMAD6, 780M) with llama3.1:8b under the v2
// harness (VRAM evict + unload + num_predict=256 + median-of-3), iGPU
// acceleration enabled (OLLAMA_IGPU_ENABLE — the config PR #1074 ships fleet-
// wide). Must stay byte-identical to the leaderboard's score_service.ts. Final
// confirm against the exact shipping Ollama config before v1.34.0 GA.
ai_tokens_per_second: 13.2,
cpu_events_multi: 19231, // all-threads on the reference box (16T)
cpu_events_single: 2351, // 1T
memory_ops_per_sec: 26862960, // 4T, 1K block (bandwidth peak)