diff --git a/CHANGELOG.md b/CHANGELOG.md index 21544fa..d6a709d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,15 @@ of current runtime behavior, see [`docs/CURRENT-STATE.md`](docs/CURRENT-STATE.md ## [Unreleased] +### Added + +- Local voice provider: the mic drives the same 28 voice tools without an + OpenAI key. The browser transcribes and speaks (Web Speech API); a local + Ollama model or the Anthropic API picks the tool calls through + `/api/local-voice/turn`. Configure with `OLLAMA_URL`/`OLLAMA_MODEL` or + `ANTHROPIC_API_KEY` (see `.env.example`). The voice instructions are now one + shared constant (`GEV_VOICE_INSTRUCTIONS`) so providers cannot drift. + ### Fixed - Mapped-site outages show their scheduled retry countdown and distinguish diff --git a/README.md b/README.md index 86f69c5..54bd161 100644 --- a/README.md +++ b/README.md @@ -227,6 +227,8 @@ The cockpit even carries its own briefing strip: nearby live signals, regional h Click **GEV MIC**, grant the microphone, and just talk. This is more than a voice-controlled remote: +> **No OpenAI key? Use the local voice provider.** Set `OLLAMA_URL=http://127.0.0.1:11434` (and `OLLAMA_MODEL`, any tool-capable model such as `qwen3`) or `ANTHROPIC_API_KEY` in `.env` and restart. The mic button switches to the local controller automatically: your browser does speech-to-text and text-to-speech (Web Speech API β€” Safari/Chrome, HTTPS required on iOS), the model only chooses which of the same 28 tools to call, and the tier chip shows the provider (`QWEN3`, `CLAUDE`) with `$0` instead of a meter. Typed commands work the same way. See `LOCAL_VOICE_*` in `.env.example`. + - **🧠 It knows what it's looking at.** The agent pulls live scene context before answering β€” including coordinates, street names, active layers, and view scale. Ask *"what city is this?"* mid-flight and it knows. - **🎯 Entity Q&A.** Click any plane, ship, or datacenter and ask *"what's this?"* It answers using the object's live telemetry. - **πŸ‘οΈ Visual grounding.** At street level, it reads a viewport screenshot to identify legible signage and building names, and is instructed never to hallucinate labels. @@ -382,6 +384,7 @@ Six keys. Four have a free tier, and the two πŸ”΄ ones are metered: | 🟑 | **Cesium ion** | πŸ—ΊοΈ Google Photorealistic 3D, world terrain, and additional ion-hosted imagery stacks. The free Community plan is for eligible individual, personal/non-commercial use and has quotas | [cesium.com/ion](https://cesium.com/ion) β€” use a public `assets:read` token and check current [pricing/eligibility](https://cesium.com/platform/cesium-ion/pricing/) | | πŸ”΄ | **Google Maps** | Direct Google Photorealistic 3D + Google place search ([Map Tiles API](https://developers.google.com/maps/documentation/tile)) | [Google Cloud Console](https://console.cloud.google.com/) β€” URL-restrict it | | πŸ”΄ | **OpenAI** | πŸŽ™οΈ The voice experience + AI HUD summary. The mini model works; the standard model is noticeably smarter. Want Gemini or another provider behind the mic? PRs welcome | [platform.openai.com](https://platform.openai.com) β€” metered, see costs below | +| 🟒 | **Local voice** | πŸŽ™οΈ The same mic and 28 tools without OpenAI: the browser transcribes and speaks (Web Speech API, HTTPS on iOS) and a local **Ollama** model or the **Anthropic** API picks the tool calls. Set `OLLAMA_URL` (+ `OLLAMA_MODEL`) or `ANTHROPIC_API_KEY` in `.env` | [ollama.com](https://ollama.com) β€” free, local Β· [console.anthropic.com](https://console.anthropic.com) β€” metered text tokens | | 🟑 | **AISStream** | 🚒 Live global ships | [aisstream.io](https://aisstream.io) β€” free signup | | 🟑 | **NASA FIRMS** | πŸ”₯ Live active fires | [firms.modaps.eosdis.nasa.gov](https://firms.modaps.eosdis.nasa.gov/api/map_key/) β€” free | | 🟑 | **TomTom** | 🚦 Live flow speeds and congestion colors for the simulated traffic layer | [developer.tomtom.com](https://developer.tomtom.com) β€” free tier available | diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 1f4d861..0657de5 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -2208,6 +2208,12 @@ silently demoting every later lookup for the session. - Ion stacks remain visible and keyboard-focusable when no ion token is configured, but expose `aria-disabled="true"` and do not switch. Their accessible label and tooltip quote `getStacks().unavailableReason` β€” the same string `setStack()` puts in the toast. OSM works keyless. The `ION` badge is gated on the stack's own `requiresIon`, so a `photoreal` chip unavailable because the Google tileset failed says so instead of falsely demanding an ion token. - Stack choice participates in share links (`src/sharelink.js`) and falls back to the best available stack when the requested one is unavailable (keyless boots land on Esri; OSM takes over automatically if Esri is unreachable). Share-link restore, the `set_map_stack` voice tool, and the chip row all land on the same `_setMapStack()` path. +### Local voice provider (September 2026) + +- `server/localVoice.js` installs `GET /api/local-voice/status` and `POST /api/local-voice/turn` next to the Realtime token route. The turn endpoint takes a provider-neutral transcript (`user` / `assistant`+`toolCalls` / `tool` entries), sends it with the shared `GEV_VOICE_INSTRUCTIONS` and `GEV_REALTIME_TOOLS` to Ollama (`/api/chat`, tools) or Anthropic (`/v1/messages`, tools), and returns `{ text, toolCalls:[{id,name,args}] }`. Provider selection: `LOCAL_VOICE_PROVIDER=auto|ollama|anthropic`; `auto` prefers Anthropic when `ANTHROPIC_API_KEY` is set, else Ollama when `OLLAMA_URL` is set, else the endpoints report `configured:false`. +- `src/voice/localVoice.js` β€” `LocalVoiceController` mirrors the surface `initGevVoiceCommands` and the voice UI use (`start/stop/isActive/sendTextCommand/notifyMapEvent/syncCostUi/bindPushToTalkShortcut/toggleVoiceTier/getDiagnostics`). Ear = Web Speech API `SpeechRecognition` (continuous, restarted after each end event, paused while the app speaks); brain = the turn endpoint with a tool loop (max 4 rounds) executed through the SAME `createGevActionRunner`; mouth = `speechSynthesis`. Needs a secure context for the microphone. +- `initGevVoiceCommands` still builds the OpenAI controller first, then probes `/api/local-voice/status`; when configured and the browser can transcribe, it rebinds the mic button to the local controller. No schema change: `GEV_REALTIME_TOOLS` is byte-identical; only the instruction text moved into `GEV_VOICE_INSTRUCTIONS`. + ### Voice Map Whiteboard / Annotations (June 2026) - Runtime entry: `src/main.js` calls `initAnnotations({ viewer, tileset })`, exposes `window.__gevAnnotations`, and passes the engine into the voice action runner. diff --git a/server/localVoice.js b/server/localVoice.js index 3ecb953..514e99f 100644 --- a/server/localVoice.js +++ b/server/localVoice.js @@ -90,7 +90,7 @@ export function sanitizeHistory(messages) { return clean.slice(-MAX_HISTORY); } -function toOllamaMessages(instructions, history) { +export function toOllamaMessages(instructions, history) { const out = [{ role: 'system', content: instructions }]; for (const m of history) { if (m.role === 'assistant') { @@ -106,7 +106,7 @@ function toOllamaMessages(instructions, history) { return out; } -function toAnthropicMessages(history) { +export function toAnthropicMessages(history) { const out = []; let pendingResults = null; const flushResults = () => { diff --git a/src/voice/localVoiceServer.test.mjs b/src/voice/localVoiceServer.test.mjs new file mode 100644 index 0000000..e7662a5 --- /dev/null +++ b/src/voice/localVoiceServer.test.mjs @@ -0,0 +1,143 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { + installLocalVoiceMiddleware, + resolveLocalVoiceProvider, + sanitizeHistory, + toAnthropicMessages, + toAnthropicTools, + toOllamaMessages, + toOllamaTools, +} from '../../server/localVoice.js'; + +const TOOLS = [ + { type: 'function', name: 'fly_to_location', description: 'Fly', parameters: { type: 'object', properties: { query: { type: 'string' } } } }, + { type: 'function', name: 'zoom_to_globe', description: 'Globe' }, +]; + +test('provider resolution: auto prefers Anthropic, then Ollama, else unconfigured', () => { + assert.equal(resolveLocalVoiceProvider({}).configured, false); + assert.equal(resolveLocalVoiceProvider({ OLLAMA_URL: 'http://x:11434/' }).provider, 'ollama'); + assert.equal(resolveLocalVoiceProvider({ OLLAMA_URL: 'http://x:11434/' }).url, 'http://x:11434'); + assert.equal(resolveLocalVoiceProvider({ OLLAMA_URL: 'http://x', ANTHROPIC_API_KEY: 'k' }).provider, 'anthropic'); + assert.equal(resolveLocalVoiceProvider({ LOCAL_VOICE_PROVIDER: 'ollama', ANTHROPIC_API_KEY: 'k' }).provider, 'ollama'); + assert.equal(resolveLocalVoiceProvider({ LOCAL_VOICE_PROVIDER: 'anthropic' }).configured, false); + assert.equal(resolveLocalVoiceProvider({ OLLAMA_URL: 'http://x', OLLAMA_MODEL: 'marcel', LOCAL_VOICE_LANG: 'es-ES' }).model, 'marcel'); + assert.equal(resolveLocalVoiceProvider({ OLLAMA_URL: 'http://x', LOCAL_VOICE_LANG: 'es-ES' }).lang, 'es-ES'); +}); + +test('tool schemas convert without touching the Realtime definitions', () => { + const ollama = toOllamaTools(TOOLS); + assert.equal(ollama[0].type, 'function'); + assert.equal(ollama[0].function.name, 'fly_to_location'); + assert.deepEqual(ollama[0].function.parameters, TOOLS[0].parameters); + assert.deepEqual(ollama[1].function.parameters, { type: 'object', properties: {} }); + const anthropic = toAnthropicTools(TOOLS); + assert.deepEqual(anthropic[0], { name: 'fly_to_location', description: 'Fly', input_schema: TOOLS[0].parameters }); + assert.equal(TOOLS[0].function, undefined, 'source tool objects must not be mutated'); +}); + +test('history sanitizer normalizes roles, tool calls, and caps length', () => { + const clean = sanitizeHistory([ + null, + { role: 'user', content: 'hola' }, + { role: 'assistant', content: '', toolCalls: [{ name: 'zoom_to_globe' }, { id: 'c2', name: 'fly_to_location', args: { query: 'Moraira' } }, { bogus: true }] }, + { role: 'tool', id: 'c2', name: 'fly_to_location', content: '{"ok":true}' }, + { role: 'system', content: 'ignored role becomes user' }, + ]); + assert.equal(clean.length, 4); + assert.deepEqual(clean[1].toolCalls, [ + { id: 'call_0', name: 'zoom_to_globe', args: {} }, + { id: 'c2', name: 'fly_to_location', args: { query: 'Moraira' } }, + ]); + assert.equal(clean[3].role, 'user'); + const long = sanitizeHistory(Array.from({ length: 60 }, (_, i) => ({ role: 'user', content: String(i) }))); + assert.equal(long.length, 24); + assert.equal(long[0].content, '36'); +}); + +test('Anthropic messages group tool results after the assistant tool_use turn and start with user', () => { + const history = sanitizeHistory([ + { role: 'user', content: 'llΓ©vame a Moraira' }, + { role: 'assistant', content: '', toolCalls: [{ id: 'c1', name: 'fly_to_location', args: { query: 'Moraira' } }, { id: 'c2', name: 'zoom_to_globe', args: {} }] }, + { role: 'tool', id: 'c1', name: 'fly_to_location', content: '{"ok":true}' }, + { role: 'tool', id: 'c2', name: 'zoom_to_globe', content: '{"ok":true}' }, + ]); + const messages = toAnthropicMessages(history); + assert.deepEqual(messages.map((m) => m.role), ['user', 'assistant', 'user']); + assert.equal(messages[1].content.filter((b) => b.type === 'tool_use').length, 2); + assert.deepEqual(messages[2].content.map((b) => b.tool_use_id), ['c1', 'c2']); + const ollama = toOllamaMessages('SYS', history); + assert.equal(ollama[0].role, 'system'); + assert.equal(ollama[2].tool_calls.length, 2); + assert.equal(ollama[3].role, 'tool'); +}); + +function fakeApp() { + const routes = new Map(); + return { use: (path, handler) => routes.set(path, handler), routes }; +} + +function fakeReq(method, body) { + const chunks = body === undefined ? [] : [Buffer.from(JSON.stringify(body))]; + return { method, url: '/', [Symbol.asyncIterator]: async function* () { for (const c of chunks) yield c; } }; +} + +function fakeRes() { + const res = { statusCode: 200, headers: {}, body: '' }; + res.setHeader = (k, v) => { res.headers[k] = v; }; + res.end = (payload) => { res.body = payload; res.done = true; }; + return res; +} + +test('status endpoint reports the resolved provider; turn refuses GET and unconfigured servers', async () => { + const app = fakeApp(); + installLocalVoiceMiddleware(app, { tools: TOOLS, instructions: 'SYS', env: {} }); + const status = fakeRes(); + app.routes.get('/api/local-voice/status')(fakeReq('GET'), status); + assert.deepEqual(JSON.parse(status.body), { configured: false, provider: null, model: null, lang: null }); + const get = fakeRes(); + await app.routes.get('/api/local-voice/turn')(fakeReq('GET'), get); + assert.equal(get.statusCode, 405); + const post = fakeRes(); + await app.routes.get('/api/local-voice/turn')(fakeReq('POST', { messages: [{ role: 'user', content: 'hi' }] }), post); + assert.equal(post.statusCode, 503); +}); + +test('turn endpoint normalizes Ollama tool calls and Anthropic tool_use blocks', async () => { + const originalFetch = globalThis.fetch; + const seen = []; + try { + globalThis.fetch = async (url, init) => { + seen.push({ url: String(url), body: JSON.parse(init.body) }); + if (String(url).includes('/api/chat')) { + return new Response(JSON.stringify({ model: 'marcel', message: { role: 'assistant', content: '', tool_calls: [{ function: { name: 'fly_to_location', arguments: { query: 'Moraira' } } }] } }), { status: 200 }); + } + return new Response(JSON.stringify({ model: 'claude', content: [{ type: 'text', text: 'Flying to Moraira.' }, { type: 'tool_use', id: 'toolu_1', name: 'fly_to_location', input: { query: 'Moraira' } }] }), { status: 200 }); + }; + const app = fakeApp(); + installLocalVoiceMiddleware(app, { tools: TOOLS, instructions: 'SYS', env: { OLLAMA_URL: 'http://ollama:11434', OLLAMA_MODEL: 'marcel' } }); + const res = fakeRes(); + await app.routes.get('/api/local-voice/turn')(fakeReq('POST', { messages: [{ role: 'user', content: 'llΓ©vame a Moraira' }] }), res); + const turn = JSON.parse(res.body); + assert.equal(res.statusCode, 200); + assert.equal(turn.provider, 'ollama'); + assert.equal(turn.toolCalls[0].name, 'fly_to_location'); + assert.deepEqual(turn.toolCalls[0].args, { query: 'Moraira' }); + assert.equal(seen[0].body.tools[0].function.name, 'fly_to_location'); + assert.equal(seen[0].body.messages[0].role, 'system'); + assert.match(seen[0].body.messages[0].content, /^SYS\n/); + + const app2 = fakeApp(); + installLocalVoiceMiddleware(app2, { tools: TOOLS, instructions: 'SYS', env: { ANTHROPIC_API_KEY: 'k' } }); + const res2 = fakeRes(); + await app2.routes.get('/api/local-voice/turn')(fakeReq('POST', { messages: [{ role: 'user', content: 'take me to Moraira' }] }), res2); + const turn2 = JSON.parse(res2.body); + assert.equal(turn2.provider, 'anthropic'); + assert.equal(turn2.text, 'Flying to Moraira.'); + assert.deepEqual(turn2.toolCalls, [{ id: 'toolu_1', name: 'fly_to_location', args: { query: 'Moraira' } }]); + assert.equal(seen[1].body.tools[0].input_schema.type, 'object'); + } finally { + globalThis.fetch = originalFetch; + } +});