docs+test(voice): local voice provider — README, CHANGELOG, CURRENT-STATE, unit tests
This commit is contained in:
TD
parent
ee7f77e8dc
commit
ddd8498031
|
|
@ -5,6 +5,15 @@ of current runtime behavior, see [`docs/CURRENT-STATE.md`](docs/CURRENT-STATE.md
|
|||
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
|
||||
- Local voice provider: the mic drives the same 28 voice tools without an
|
||||
OpenAI key. The browser transcribes and speaks (Web Speech API); a local
|
||||
Ollama model or the Anthropic API picks the tool calls through
|
||||
`/api/local-voice/turn`. Configure with `OLLAMA_URL`/`OLLAMA_MODEL` or
|
||||
`ANTHROPIC_API_KEY` (see `.env.example`). The voice instructions are now one
|
||||
shared constant (`GEV_VOICE_INSTRUCTIONS`) so providers cannot drift.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Mapped-site outages show their scheduled retry countdown and distinguish
|
||||
|
|
|
|||
|
|
@ -227,6 +227,8 @@ The cockpit even carries its own briefing strip: nearby live signals, regional h
|
|||
|
||||
Click **GEV MIC**, grant the microphone, and just talk. This is more than a voice-controlled remote:
|
||||
|
||||
> **No OpenAI key? Use the local voice provider.** Set `OLLAMA_URL=http://127.0.0.1:11434` (and `OLLAMA_MODEL`, any tool-capable model such as `qwen3`) or `ANTHROPIC_API_KEY` in `.env` and restart. The mic button switches to the local controller automatically: your browser does speech-to-text and text-to-speech (Web Speech API — Safari/Chrome, HTTPS required on iOS), the model only chooses which of the same 28 tools to call, and the tier chip shows the provider (`QWEN3`, `CLAUDE`) with `$0` instead of a meter. Typed commands work the same way. See `LOCAL_VOICE_*` in `.env.example`.
|
||||
|
||||
- **🧠 It knows what it's looking at.** The agent pulls live scene context before answering — including coordinates, street names, active layers, and view scale. Ask *"what city is this?"* mid-flight and it knows.
|
||||
- **🎯 Entity Q&A.** Click any plane, ship, or datacenter and ask *"what's this?"* It answers using the object's live telemetry.
|
||||
- **👁️ Visual grounding.** At street level, it reads a viewport screenshot to identify legible signage and building names, and is instructed never to hallucinate labels.
|
||||
|
|
@ -382,6 +384,7 @@ Six keys. Four have a free tier, and the two 🔴 ones are metered:
|
|||
| 🟡 | **Cesium ion** | 🗺️ Google Photorealistic 3D, world terrain, and additional ion-hosted imagery stacks. The free Community plan is for eligible individual, personal/non-commercial use and has quotas | [cesium.com/ion](https://cesium.com/ion) — use a public `assets:read` token and check current [pricing/eligibility](https://cesium.com/platform/cesium-ion/pricing/) |
|
||||
| 🔴 | **Google Maps** | Direct Google Photorealistic 3D + Google place search ([Map Tiles API](https://developers.google.com/maps/documentation/tile)) | [Google Cloud Console](https://console.cloud.google.com/) — URL-restrict it |
|
||||
| 🔴 | **OpenAI** | 🎙️ The voice experience + AI HUD summary. The mini model works; the standard model is noticeably smarter. Want Gemini or another provider behind the mic? PRs welcome | [platform.openai.com](https://platform.openai.com) — metered, see costs below |
|
||||
| 🟢 | **Local voice** | 🎙️ The same mic and 28 tools without OpenAI: the browser transcribes and speaks (Web Speech API, HTTPS on iOS) and a local **Ollama** model or the **Anthropic** API picks the tool calls. Set `OLLAMA_URL` (+ `OLLAMA_MODEL`) or `ANTHROPIC_API_KEY` in `.env` | [ollama.com](https://ollama.com) — free, local · [console.anthropic.com](https://console.anthropic.com) — metered text tokens |
|
||||
| 🟡 | **AISStream** | 🚢 Live global ships | [aisstream.io](https://aisstream.io) — free signup |
|
||||
| 🟡 | **NASA FIRMS** | 🔥 Live active fires | [firms.modaps.eosdis.nasa.gov](https://firms.modaps.eosdis.nasa.gov/api/map_key/) — free |
|
||||
| 🟡 | **TomTom** | 🚦 Live flow speeds and congestion colors for the simulated traffic layer | [developer.tomtom.com](https://developer.tomtom.com) — free tier available |
|
||||
|
|
|
|||
|
|
@ -2208,6 +2208,12 @@ silently demoting every later lookup for the session.
|
|||
- Ion stacks remain visible and keyboard-focusable when no ion token is configured, but expose `aria-disabled="true"` and do not switch. Their accessible label and tooltip quote `getStacks().unavailableReason` — the same string `setStack()` puts in the toast. OSM works keyless. The `ION` badge is gated on the stack's own `requiresIon`, so a `photoreal` chip unavailable because the Google tileset failed says so instead of falsely demanding an ion token.
|
||||
- Stack choice participates in share links (`src/sharelink.js`) and falls back to the best available stack when the requested one is unavailable (keyless boots land on Esri; OSM takes over automatically if Esri is unreachable). Share-link restore, the `set_map_stack` voice tool, and the chip row all land on the same `_setMapStack()` path.
|
||||
|
||||
### Local voice provider (September 2026)
|
||||
|
||||
- `server/localVoice.js` installs `GET /api/local-voice/status` and `POST /api/local-voice/turn` next to the Realtime token route. The turn endpoint takes a provider-neutral transcript (`user` / `assistant`+`toolCalls` / `tool` entries), sends it with the shared `GEV_VOICE_INSTRUCTIONS` and `GEV_REALTIME_TOOLS` to Ollama (`/api/chat`, tools) or Anthropic (`/v1/messages`, tools), and returns `{ text, toolCalls:[{id,name,args}] }`. Provider selection: `LOCAL_VOICE_PROVIDER=auto|ollama|anthropic`; `auto` prefers Anthropic when `ANTHROPIC_API_KEY` is set, else Ollama when `OLLAMA_URL` is set, else the endpoints report `configured:false`.
|
||||
- `src/voice/localVoice.js` — `LocalVoiceController` mirrors the surface `initGevVoiceCommands` and the voice UI use (`start/stop/isActive/sendTextCommand/notifyMapEvent/syncCostUi/bindPushToTalkShortcut/toggleVoiceTier/getDiagnostics`). Ear = Web Speech API `SpeechRecognition` (continuous, restarted after each end event, paused while the app speaks); brain = the turn endpoint with a tool loop (max 4 rounds) executed through the SAME `createGevActionRunner`; mouth = `speechSynthesis`. Needs a secure context for the microphone.
|
||||
- `initGevVoiceCommands` still builds the OpenAI controller first, then probes `/api/local-voice/status`; when configured and the browser can transcribe, it rebinds the mic button to the local controller. No schema change: `GEV_REALTIME_TOOLS` is byte-identical; only the instruction text moved into `GEV_VOICE_INSTRUCTIONS`.
|
||||
|
||||
### Voice Map Whiteboard / Annotations (June 2026)
|
||||
|
||||
- Runtime entry: `src/main.js` calls `initAnnotations({ viewer, tileset })`, exposes `window.__gevAnnotations`, and passes the engine into the voice action runner.
|
||||
|
|
|
|||
|
|
@ -90,7 +90,7 @@ export function sanitizeHistory(messages) {
|
|||
return clean.slice(-MAX_HISTORY);
|
||||
}
|
||||
|
||||
function toOllamaMessages(instructions, history) {
|
||||
export function toOllamaMessages(instructions, history) {
|
||||
const out = [{ role: 'system', content: instructions }];
|
||||
for (const m of history) {
|
||||
if (m.role === 'assistant') {
|
||||
|
|
@ -106,7 +106,7 @@ function toOllamaMessages(instructions, history) {
|
|||
return out;
|
||||
}
|
||||
|
||||
function toAnthropicMessages(history) {
|
||||
export function toAnthropicMessages(history) {
|
||||
const out = [];
|
||||
let pendingResults = null;
|
||||
const flushResults = () => {
|
||||
|
|
|
|||
|
|
@ -0,0 +1,143 @@
|
|||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import {
|
||||
installLocalVoiceMiddleware,
|
||||
resolveLocalVoiceProvider,
|
||||
sanitizeHistory,
|
||||
toAnthropicMessages,
|
||||
toAnthropicTools,
|
||||
toOllamaMessages,
|
||||
toOllamaTools,
|
||||
} from '../../server/localVoice.js';
|
||||
|
||||
const TOOLS = [
|
||||
{ type: 'function', name: 'fly_to_location', description: 'Fly', parameters: { type: 'object', properties: { query: { type: 'string' } } } },
|
||||
{ type: 'function', name: 'zoom_to_globe', description: 'Globe' },
|
||||
];
|
||||
|
||||
test('provider resolution: auto prefers Anthropic, then Ollama, else unconfigured', () => {
|
||||
assert.equal(resolveLocalVoiceProvider({}).configured, false);
|
||||
assert.equal(resolveLocalVoiceProvider({ OLLAMA_URL: 'http://x:11434/' }).provider, 'ollama');
|
||||
assert.equal(resolveLocalVoiceProvider({ OLLAMA_URL: 'http://x:11434/' }).url, 'http://x:11434');
|
||||
assert.equal(resolveLocalVoiceProvider({ OLLAMA_URL: 'http://x', ANTHROPIC_API_KEY: 'k' }).provider, 'anthropic');
|
||||
assert.equal(resolveLocalVoiceProvider({ LOCAL_VOICE_PROVIDER: 'ollama', ANTHROPIC_API_KEY: 'k' }).provider, 'ollama');
|
||||
assert.equal(resolveLocalVoiceProvider({ LOCAL_VOICE_PROVIDER: 'anthropic' }).configured, false);
|
||||
assert.equal(resolveLocalVoiceProvider({ OLLAMA_URL: 'http://x', OLLAMA_MODEL: 'marcel', LOCAL_VOICE_LANG: 'es-ES' }).model, 'marcel');
|
||||
assert.equal(resolveLocalVoiceProvider({ OLLAMA_URL: 'http://x', LOCAL_VOICE_LANG: 'es-ES' }).lang, 'es-ES');
|
||||
});
|
||||
|
||||
test('tool schemas convert without touching the Realtime definitions', () => {
|
||||
const ollama = toOllamaTools(TOOLS);
|
||||
assert.equal(ollama[0].type, 'function');
|
||||
assert.equal(ollama[0].function.name, 'fly_to_location');
|
||||
assert.deepEqual(ollama[0].function.parameters, TOOLS[0].parameters);
|
||||
assert.deepEqual(ollama[1].function.parameters, { type: 'object', properties: {} });
|
||||
const anthropic = toAnthropicTools(TOOLS);
|
||||
assert.deepEqual(anthropic[0], { name: 'fly_to_location', description: 'Fly', input_schema: TOOLS[0].parameters });
|
||||
assert.equal(TOOLS[0].function, undefined, 'source tool objects must not be mutated');
|
||||
});
|
||||
|
||||
test('history sanitizer normalizes roles, tool calls, and caps length', () => {
|
||||
const clean = sanitizeHistory([
|
||||
null,
|
||||
{ role: 'user', content: 'hola' },
|
||||
{ role: 'assistant', content: '', toolCalls: [{ name: 'zoom_to_globe' }, { id: 'c2', name: 'fly_to_location', args: { query: 'Moraira' } }, { bogus: true }] },
|
||||
{ role: 'tool', id: 'c2', name: 'fly_to_location', content: '{"ok":true}' },
|
||||
{ role: 'system', content: 'ignored role becomes user' },
|
||||
]);
|
||||
assert.equal(clean.length, 4);
|
||||
assert.deepEqual(clean[1].toolCalls, [
|
||||
{ id: 'call_0', name: 'zoom_to_globe', args: {} },
|
||||
{ id: 'c2', name: 'fly_to_location', args: { query: 'Moraira' } },
|
||||
]);
|
||||
assert.equal(clean[3].role, 'user');
|
||||
const long = sanitizeHistory(Array.from({ length: 60 }, (_, i) => ({ role: 'user', content: String(i) })));
|
||||
assert.equal(long.length, 24);
|
||||
assert.equal(long[0].content, '36');
|
||||
});
|
||||
|
||||
test('Anthropic messages group tool results after the assistant tool_use turn and start with user', () => {
|
||||
const history = sanitizeHistory([
|
||||
{ role: 'user', content: 'llévame a Moraira' },
|
||||
{ role: 'assistant', content: '', toolCalls: [{ id: 'c1', name: 'fly_to_location', args: { query: 'Moraira' } }, { id: 'c2', name: 'zoom_to_globe', args: {} }] },
|
||||
{ role: 'tool', id: 'c1', name: 'fly_to_location', content: '{"ok":true}' },
|
||||
{ role: 'tool', id: 'c2', name: 'zoom_to_globe', content: '{"ok":true}' },
|
||||
]);
|
||||
const messages = toAnthropicMessages(history);
|
||||
assert.deepEqual(messages.map((m) => m.role), ['user', 'assistant', 'user']);
|
||||
assert.equal(messages[1].content.filter((b) => b.type === 'tool_use').length, 2);
|
||||
assert.deepEqual(messages[2].content.map((b) => b.tool_use_id), ['c1', 'c2']);
|
||||
const ollama = toOllamaMessages('SYS', history);
|
||||
assert.equal(ollama[0].role, 'system');
|
||||
assert.equal(ollama[2].tool_calls.length, 2);
|
||||
assert.equal(ollama[3].role, 'tool');
|
||||
});
|
||||
|
||||
function fakeApp() {
|
||||
const routes = new Map();
|
||||
return { use: (path, handler) => routes.set(path, handler), routes };
|
||||
}
|
||||
|
||||
function fakeReq(method, body) {
|
||||
const chunks = body === undefined ? [] : [Buffer.from(JSON.stringify(body))];
|
||||
return { method, url: '/', [Symbol.asyncIterator]: async function* () { for (const c of chunks) yield c; } };
|
||||
}
|
||||
|
||||
function fakeRes() {
|
||||
const res = { statusCode: 200, headers: {}, body: '' };
|
||||
res.setHeader = (k, v) => { res.headers[k] = v; };
|
||||
res.end = (payload) => { res.body = payload; res.done = true; };
|
||||
return res;
|
||||
}
|
||||
|
||||
test('status endpoint reports the resolved provider; turn refuses GET and unconfigured servers', async () => {
|
||||
const app = fakeApp();
|
||||
installLocalVoiceMiddleware(app, { tools: TOOLS, instructions: 'SYS', env: {} });
|
||||
const status = fakeRes();
|
||||
app.routes.get('/api/local-voice/status')(fakeReq('GET'), status);
|
||||
assert.deepEqual(JSON.parse(status.body), { configured: false, provider: null, model: null, lang: null });
|
||||
const get = fakeRes();
|
||||
await app.routes.get('/api/local-voice/turn')(fakeReq('GET'), get);
|
||||
assert.equal(get.statusCode, 405);
|
||||
const post = fakeRes();
|
||||
await app.routes.get('/api/local-voice/turn')(fakeReq('POST', { messages: [{ role: 'user', content: 'hi' }] }), post);
|
||||
assert.equal(post.statusCode, 503);
|
||||
});
|
||||
|
||||
test('turn endpoint normalizes Ollama tool calls and Anthropic tool_use blocks', async () => {
|
||||
const originalFetch = globalThis.fetch;
|
||||
const seen = [];
|
||||
try {
|
||||
globalThis.fetch = async (url, init) => {
|
||||
seen.push({ url: String(url), body: JSON.parse(init.body) });
|
||||
if (String(url).includes('/api/chat')) {
|
||||
return new Response(JSON.stringify({ model: 'marcel', message: { role: 'assistant', content: '', tool_calls: [{ function: { name: 'fly_to_location', arguments: { query: 'Moraira' } } }] } }), { status: 200 });
|
||||
}
|
||||
return new Response(JSON.stringify({ model: 'claude', content: [{ type: 'text', text: 'Flying to Moraira.' }, { type: 'tool_use', id: 'toolu_1', name: 'fly_to_location', input: { query: 'Moraira' } }] }), { status: 200 });
|
||||
};
|
||||
const app = fakeApp();
|
||||
installLocalVoiceMiddleware(app, { tools: TOOLS, instructions: 'SYS', env: { OLLAMA_URL: 'http://ollama:11434', OLLAMA_MODEL: 'marcel' } });
|
||||
const res = fakeRes();
|
||||
await app.routes.get('/api/local-voice/turn')(fakeReq('POST', { messages: [{ role: 'user', content: 'llévame a Moraira' }] }), res);
|
||||
const turn = JSON.parse(res.body);
|
||||
assert.equal(res.statusCode, 200);
|
||||
assert.equal(turn.provider, 'ollama');
|
||||
assert.equal(turn.toolCalls[0].name, 'fly_to_location');
|
||||
assert.deepEqual(turn.toolCalls[0].args, { query: 'Moraira' });
|
||||
assert.equal(seen[0].body.tools[0].function.name, 'fly_to_location');
|
||||
assert.equal(seen[0].body.messages[0].role, 'system');
|
||||
assert.match(seen[0].body.messages[0].content, /^SYS\n/);
|
||||
|
||||
const app2 = fakeApp();
|
||||
installLocalVoiceMiddleware(app2, { tools: TOOLS, instructions: 'SYS', env: { ANTHROPIC_API_KEY: 'k' } });
|
||||
const res2 = fakeRes();
|
||||
await app2.routes.get('/api/local-voice/turn')(fakeReq('POST', { messages: [{ role: 'user', content: 'take me to Moraira' }] }), res2);
|
||||
const turn2 = JSON.parse(res2.body);
|
||||
assert.equal(turn2.provider, 'anthropic');
|
||||
assert.equal(turn2.text, 'Flying to Moraira.');
|
||||
assert.deepEqual(turn2.toolCalls, [{ id: 'toolu_1', name: 'fly_to_location', args: { query: 'Moraira' } }]);
|
||||
assert.equal(seen[1].body.tools[0].input_schema.type, 'object');
|
||||
} finally {
|
||||
globalThis.fetch = originalFetch;
|
||||
}
|
||||
});
|
||||
Loading…
Reference in New Issue