project-nomad/admin/app
chriscrosstalk d1535d17b9
fix(benchmark): block leaderboard submission when AI runs on a remote host (#1157)
DockerService.getServiceURL() resolves ai.remoteOllamaUrl ahead of the local
container, so when a remote AI host is configured the AI channel measures THAT
machine while every other channel measures this one. The submission then reports
someone else's tok/s under this hardware's CPU, RAM and disk.

Under v2 this matters more than it did under v1: ai_tokens_per_second carries
0.30 of the weight and the score is uncapped, so a remote GPU's throughput is no
longer limited by a clamp.

Adds a guard alongside the existing submit-time checks. Uses the same truthiness
predicate as getServiceURL, so the guard fires exactly when the remote routing
it is guarding against would occur. KVStore.clearValue() nulls the value and
getValue() returns null for that, so a cleared key correctly does not trip it.

Blocks submission only. Running the benchmark locally is still useful to the
operator — it just isn't a result about this box, so it shouldn't go on a board
that ranks hardware.

Known limitation: the check reads the KV at submit time, not at measurement
time, so benchmarking with a remote host and then clearing the setting before
submitting would still get through. That is a deliberate workaround rather than
an accident, and closing it properly needs a column on benchmark_results to
record how inference was reached. Worth doing if it ever shows up in practice;
not worth a migration on the evidence available.

Refs #1151
2026-07-27 10:17:14 -07:00
..
controllers fix(UI): add API reference to Settings sidebar 2026-07-24 03:46:30 +00:00
data refactor(drug-reference): make collections JSON the single source for curated data (#1130) 2026-07-21 15:50:47 -07:00
exceptions fix(Docs): documentation renderer fixes 2025-12-23 16:00:33 -08:00
jobs Add offline FDA drug reference (labels, interaction view, conditions, remedies) (#1040) 2026-07-21 11:06:18 -07:00
middleware fix(API): skip compression for Server-Sent Events (#798) 2026-05-20 10:16:00 -07:00
models Add offline FDA drug reference (labels, interaction view, conditions, remedies) (#1040) 2026-07-21 11:06:18 -07:00
services fix(benchmark): block leaderboard submission when AI runs on a remote host (#1157) 2026-07-27 10:17:14 -07:00
utils fix(amd): coerce gfx1103 (780M) to HSA_OVERRIDE 11.0.0 so it stays on GPU (#1134) 2026-07-22 11:07:42 -07:00
validators feat: auto-generating OpenAPI docs with Scalar UI (#1128) 2026-07-21 12:36:40 -07:00