diff --git a/CHANGELOG.md b/CHANGELOG.md index 21544fa..2562818 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,12 @@ of current runtime behavior, see [`docs/CURRENT-STATE.md`](docs/CURRENT-STATE.md ### Fixed +- Street Traffic no longer remains in `SYNCING ROAD NETWORK` when the public + Overpass mirrors are unavailable. Exact bounded traffic road queries now use + OpenStreetMap's standard map endpoint first, translate its XML response into + the existing Overpass-shaped payload, and retain the generic Overpass mirror + rotation as a fallback. TomTom flow tiles and their attribution remain + unchanged. - Mapped-site outages show their scheduled retry countdown and distinguish known Overpass rate limits, timeouts, and query failures. Search feedback no longer claims a refresh succeeded while the layer is unavailable or loading. diff --git a/DATA_SOURCES.md b/DATA_SOURCES.md index faa351d..59b7544 100644 --- a/DATA_SOURCES.md +++ b/DATA_SOURCES.md @@ -24,7 +24,7 @@ How to read this: | **The Space Devs — Launch Library 2 v2.3** | Recent launch, payload, stage, and recovery metadata for Space Missions (30d) | [The Space Devs terms of use](https://github.com/TheSpaceDevs/Tutorials/blob/main/faqs/faq_TSD.md#terms-of-use): data may be used and shared in any form; avoid forwarding it without added value; attribution is encouraged (not mandatory). [Official API limits](https://ll.thespacedevs.com/docs/): 15 unauthenticated calls/hour; optional token | "Launch Library 2 — The Space Devs" (courtesy attribution) | | **Esri World Imagery** (ArcGIS Online tile service) | The keyless satellite basemap — the default landing when no Google/ion credential is configured, and the "Esri Satellite" map stack | [Esri Master Agreement](https://www.esri.com/en-us/legal/terms/full-master-agreement): the public World Imagery service is usable in public-facing apps with attribution; no key is required for this classic endpoint, but Esri governs and can change access — an app at scale should review current ArcGIS Location Platform terms | "Powered by Esri — Source: Esri, Maxar, Earthstar Geographics, and the GIS User Community" (provider carries the service's own credit line) | | **USGS** | Earthquakes | U.S. public domain | "Data courtesy of the U.S. Geological Survey" | -| **OpenStreetMap (Overpass API)** | Road geometry for traffic | ODbL 1.0 | "© OpenStreetMap contributors" | +| **OpenStreetMap (Standard Map API + Overpass API)** | Road geometry for traffic | ODbL 1.0 | "© OpenStreetMap contributors" | | **TomTom Traffic API** (flow vector tiles) | Live congestion coloring for the traffic layer (optional, BYOK) | [TomTom for Developers terms](https://developer.tomtom.com) (proprietary, your own key; free tier currently 200K tile requests/month — see [current pricing](https://docs.tomtom.com/pricing/)) | "Traffic flow data © TomTom" — registered when live mode activates | | **OpenStreetMap (Overpass API)** | Viewport-bounded mapped installation context for Global Context | ODbL 1.0 | "© OpenStreetMap contributors" (incomplete mapped context) | | **OpenStreetMap (Nominatim)** | Reverse-geocoded place label in the cockpit Local Info page | ODbL 1.0 + Nominatim usage policy | "© OpenStreetMap contributors" | @@ -47,6 +47,7 @@ How to read this: - **TfL JamCams.** The camera list comes from the keyless `api.tfl.gov.uk` endpoint (an optional `TFL_APP_KEY` raises its rate limit); frames come from TfL's public S3 bucket. The "Powered by TfL Open Data" attribution is required by TfL's terms and is registered in the Data attribution popover. - **Radio Browser.** `/api/radio/stations` discovers official API mirrors, makes bounded and coalesced healthy/geolocated HTTPS-station queries, caches the normalized public-domain directory for 45 minutes, and may serve the last good catalog for up to seven days during an outage. Refreshes must meet minimum accepted-query and station coverage before replacing a warm catalog; schema-valid responses whose rows all fail the product's health policy do not count as successful queries. A usable partial cold catalog is explicitly `DEGRADED`, and malformed or empty successful payloads are rejected atomically. Every directory and click-count request rejects redirects, validates all resolved addresses as globally routable (including reserved/documentation IPv4 and special/non-global IPv6 exclusions), and pins the TLS connection to a validated address. Only MP3/AAC non-HLS directory rows with public HTTPS stream targets are returned; favicons are intentionally omitted. Pressing play connects one browser audio element directly to the selected broadcaster and calls the directory's click counter through known-ID-only `POST /api/radio/click/:uuid`. GEV never proxies, caches, records, bundles, or redistributes audio. Radio Browser supplies station-level tags, not dependable current-song or upcoming-program metadata, so Radio filtering never claims either. Direct playback exposes the listener's IP address to the broadcaster, whose own stream terms apply. - **TomTom Traffic.** Optional and BYOK: without `TOMTOM_API_KEY` the traffic layer runs its built-in simulation and no TomTom data (or attribution) appears. With a key, flow vector tiles are fetched through the server-side `/api/tomtom` proxy (120 s cache + a daily tile-budget governor — `TOMTOM_DAILY_TILE_BUDGET`, default 40,000, a configurable application safety ceiling, not a guarantee of staying within TomTom's monthly free allowance; TomTom's [current pricing](https://docs.tomtom.com/pricing/) lists 200K free tile requests per month) and the "Traffic flow data © TomTom" credit is registered in the Data attribution popover the moment live mode activates. TomTom data is served live and cached only transiently (≤120 s TTL under `.gev-cache/`, gitignored) — it is not bundled or redistributed. One 23 KB point-in-time tile snapshot is committed as a decode-test fixture (`src/data/fixtures/`, © TomTom, never served to the app). +- **OpenStreetMap road geometry.** Street Traffic's exact, bounded highway query uses the standard `/api/0.6/map` endpoint first, parses its XML response server-side into the existing Overpass-shaped road payload, and keeps the established four-mirror Overpass rotation as a fallback. Other Overpass-backed features continue to use the generic query path. Road geometry is fetched at runtime, is not bundled or redistributed, and remains covered by the in-app "© OpenStreetMap contributors" attribution. - **Re:Earth Terrain.** Keyless (no API key). Used two ways: (1) `src/mapStackController.js` swaps in a `Cesium.CesiumTerrainProvider` pointed at Re:Earth's `cesium-mesh/ellipsoid` quantized-mesh endpoint for globe stacks without a Cesium ion token (e.g. OSM), replacing a flat `EllipsoidTerrainProvider`; falls back to the flat provider if the endpoint can't be reached. (2) The server-side `/api/terrain/heights` proxy (disk-cached, serve-stale) resolves per-point ellipsoidal ground height for entity placement. Both are best-effort with a keyless-safe fallback (bundled EGM96 geoid math) if Re:Earth is unreachable. - **Global Context installation context.** `/api/military-installations` queries only an allow-listed subset of OSM `military=*` and `landuse=military` features inside a maximum 10° non-dateline viewport. It caches and may serve stale mapped context, but it is neither a global installation database nor evidence of capability, activity, or absence. User-requested Google Places results remain separately sourced candidates unless their returned types explicitly establish military classification; generic offices, museums, and similarly ambiguous matches are excluded from military proximity counts. - **Cockpit regional briefing.** `/api/regional-brief` rounds aircraft coordinates into 0.1° cache cells, caches results for five minutes, and serializes Nominatim calls at no more than one request per second. Google News RSS is queried with the resolved locality/region first; GDELT is used only when that RSS query fails or is empty. Google's published Google News terms restrict that source to personal, noncommercial use, so commercial deployments must disable/replace it or obtain separate permission; GDELT permits commercial dataset use with citation. The Data attribution popover identifies the active headline sources; article links retain publisher attribution. Headlines are location-query matches, not verified incidents, risk rankings, or evidence that a location is safe. Empty, partial, stale, and unavailable source states remain distinct. Open-Meteo supplies current conditions independently of the news source. `WX OFF` disables cockpit weather rendering only; the Local Info briefing still fetches its source-backed weather values and displays the required linked Open-Meteo credit. diff --git a/docs/CURRENT-STATE.md b/docs/CURRENT-STATE.md index 1f4d861..ff50607 100644 --- a/docs/CURRENT-STATE.md +++ b/docs/CURRENT-STATE.md @@ -2030,6 +2030,7 @@ silently demoting every later lookup for the session. - `/api/overpass` fans out across four public mirrors. `overpassPayloadIsData()` governs cache reads, writes, and stale fallback: only a 2xx that is neither rate-limited nor a body-level runtime error qualifies. Previously stored refusals are ignored on both fresh and stale reads, so upgrading does not require manually clearing the disk cache. - HTTP refusals such as 406 now rotate alongside the existing network, rate-limit, and runtime-error cases. A refusal from one mirror no longer prevents reaching healthy alternatives or persists under the seven-day road/month-long boundary cache TTLs. Concurrent identical queries share one mirror sequence; if it fails, both the initiating and joined callers can use the same last-good data. - A refusal every mirror agrees on is still reported with the first mirror's status and body, so a genuinely malformed query says what upstream said — but only after every mirror has had the chance to answer it. `fetchOverpassPayload` takes injectable endpoints and fetch so the rotation is tested without a live mirror (`src/overpassProxy.test.mjs`). +- Street Traffic's exact bounded highway queries take a separate 12-second path through OpenStreetMap's standard `/api/0.6/map` endpoint. The server parses its XML nodes, ways, and tags into the existing Overpass-shaped road payload, so `traffic.js` and TomTom matching do not need a second client contract. If the OSM map request fails — or the query is not the traffic shape — the established four-mirror Overpass path remains the fallback; generic Overpass-backed features are unchanged. ### Share-link v2 layer state (August 2026) @@ -2543,10 +2544,11 @@ easier to meet (detection is now on more often), but does not create it. is configured (env or Keychain `tomtom-api`/`api-key`), which enables `live` mode: TomTom flow vector tiles via the budget-governed `/api/tomtom` proxy (`.gev-cache/tomtom/`, 120 s TTL, `TOMTOM_DAILY_TILE_BUDGET` default 40k/day), - decoded client-side (`flowTiles.js`), matched onto Overpass roads + decoded client-side (`flowTiles.js`), matched onto OSM roads (`flowMatch.js`), and rendered as green/amber/red dot color + speed/density scaling (`trafficFlowStyle.js`); closures spawn no dots; unmatched roads stay - white. Road fetch bounds center on the camera look-at point (`trafficBounds.js`). + white. Road fetch bounds center on the camera look-at point (`trafficBounds.js`); + the proxy uses the bounded OSM map path before falling back to Overpass. - Development captures opened with `?trafficDebug=1` mint an interaction anchor from the exact `camera.changed` event that arms each debounced load, then emit scheduling-correlated User Timing entries for production `response.json`, road diff --git a/src/data/traffic.js b/src/data/traffic.js index 484f599..3f4ec7e 100644 --- a/src/data/traffic.js +++ b/src/data/traffic.js @@ -18,15 +18,17 @@ import { holdContinuousRender, releaseContinuousRender } from '../renderGovernor * @file Street Traffic — animated dots along OSM road polylines, colored by * live TomTom congestion when a key is configured. * - * Road geometry: OSM Overpass API (free, no auth). Fetches road polylines for - * the camera viewport, spawns PointPrimitives that lerp along pre-computed - * Cartesian3 waypoints. Camera-gated: only active below ~8 km altitude. + * Road geometry: OpenStreetMap's standard map endpoint for bounded traffic + * queries, with the server-side Overpass proxy as fallback. Fetches road + * polylines for the camera viewport, spawns PointPrimitives that lerp along + * pre-computed Cartesian3 waypoints. Camera-gated: only active below ~8 km + * altitude. * * Two modes (decided once per session via `/api/tomtom/status`): * - `sim` (keyless default): white dots at hardcoded per-road-class speeds — * the original simulation, byte-identical behavior. * - `live`: TomTom flow tiles (`flowTiles.js`) are matched onto the same - * Overpass roads (`flowMatch.js`); matched roads color/slow/densify their + * OSM roads (`flowMatch.js`); matched roads color/slow/densify their * dots by real congestion (`trafficFlowStyle.js`), closed roads spawn no * dots, and unmatched roads keep the simulated white. * diff --git a/src/overpassProxy.test.mjs b/src/overpassProxy.test.mjs index 1069461..1012456 100644 --- a/src/overpassProxy.test.mjs +++ b/src/overpassProxy.test.mjs @@ -11,7 +11,13 @@ import { mkdir, readFile, writeFile, unlink } from 'node:fs/promises'; import { createHash, randomUUID } from 'node:crypto'; import path from 'node:path'; import { Readable } from 'node:stream'; -import createViteConfig, { fetchOverpassPayload, overpassPayloadIsData, readOverpassDisk } from '../vite.config.js'; +import createViteConfig, { + fetchOsmMapRoadPayload, + fetchOverpassPayload, + overpassPayloadIsData, + parseOsmMapRoads, + readOverpassDisk, +} from '../vite.config.js'; const ENDPOINTS = ['https://a.example/api', 'https://b.example/api', 'https://c.example/api']; @@ -39,6 +45,49 @@ const run = (byUrl) => { const DATA = { status: 200, body: '{"elements":[]}' }; +const OSM_ROAD_XML = ` + + + + + + + + + + + + +`; + +test('OSM map XML converts referenced nodes into traffic road geometry', () => { + const payload = parseOsmMapRoads(OSM_ROAD_XML, new Set(['primary'])); + assert.equal(payload.elements.length, 1); + assert.equal(payload.elements[0].id, '101'); + assert.equal(payload.elements[0].tags.highway, 'primary'); + assert.equal(payload.elements[0].tags.oneway, 'yes'); + assert.deepEqual(payload.elements[0].geometry, [ + { lat: -3.76, lon: -38.49 }, + { lat: -3.759, lon: -38.489 }, + { lat: -3.758, lon: -38.488 }, + ]); +}); + +test('OSM map road fallback requests the traffic bbox and returns Overpass-shaped JSON', async () => { + let requestedUrl = null; + const query = '[out:json][timeout:12];(way["highway"~"^(motorway|trunk|primary|secondary)$"](-3.7609,-38.4967,-3.7540,-38.4827););out geom qt;'; + const payload = await fetchOsmMapRoadPayload(`data=${encodeURIComponent(query)}`, 1e6, { + fetchImpl: async (url) => { + requestedUrl = String(url); + return new Response(OSM_ROAD_XML, { status: 200, headers: { 'content-type': 'application/xml' } }); + }, + }); + assert.equal(payload.status, 200); + assert.equal(payload.endpoint, 'https://api.openstreetmap.org/api/0.6/map'); + assert.match(requestedUrl, /bbox=-38\.4967%2C-3\.7609%2C-38\.4827%2C-3\.754/); + assert.equal(JSON.parse(payload.body).elements.length, 1); +}); + test('disk cache rejects old refusals for fresh and stale reads but preserves last-good data', async () => { const key = `overpass-cache-regression-${randomUUID()}`; const directory = path.join(process.cwd(), '.gev-cache', 'overpass'); diff --git a/vite.config.js b/vite.config.js index 34dac52..abf2958 100644 --- a/vite.config.js +++ b/vite.config.js @@ -198,6 +198,8 @@ const OVERPASS_UPSTREAMS = [ // Verified: planet coverage (Texas query), CORS *, ~5-20 s cold latency. 'https://overpass.private.coffee/api/interpreter', ]; +/** Standard OSM map endpoint used as a bounded road-geometry fallback. */ +const OSM_MAP_ENDPOINT = 'https://api.openstreetmap.org/api/0.6/map'; /** * TTL for FRESH cached Overpass responses (ms). Road geometry is static for * months — the original 45 s TTL forced a public-mirror round-trip on nearly @@ -221,6 +223,8 @@ const OVERPASS_BOUNDARY_DISK_TTL_MS = 30 * 86_400_000; const OVERPASS_DISK_DIR = path.join(process.cwd(), '.gev-cache', 'overpass'); /** Per-upstream fetch timeout (ms). */ const OVERPASS_TIMEOUT_MS = 22000; +/** OSM map fallback timeout (ms). */ +const OSM_MAP_TIMEOUT_MS = 12000; /** Max entries in the Overpass response cache (LRU-like, oldest evicted first). */ const OVERPASS_CACHE_MAX_ENTRIES = 120; /** @type {Map} */ @@ -2573,6 +2577,165 @@ export function overpassPayloadIsData(payload) { && !payload.runtimeError; } +const TRAFFIC_ROAD_HIGHWAYS = new Set([ + 'motorway', 'trunk', 'primary', 'secondary', + 'tertiary', 'residential', 'unclassified', +]); + +/** + * Extract the bounded road query shape emitted by src/data/traffic.js. The + * standard OSM map endpoint cannot execute arbitrary Overpass QL, so the + * fallback is deliberately limited to this exact highway+bbox form. + */ +function trafficRoadQuerySpec(body) { + let query; + try { + query = new URLSearchParams(String(body || '')).get('data'); + } catch { + return null; + } + if (!query) return null; + + const match = query.match( + /way\s*\[\s*"highway"\s*~\s*"\^\(([^\"]+)\)\$"\s*\]\s*\(\s*(-?\d+(?:\.\d+)?)\s*,\s*(-?\d+(?:\.\d+)?)\s*,\s*(-?\d+(?:\.\d+)?)\s*,\s*(-?\d+(?:\.\d+)?)\s*\)\s*;/i, + ); + if (!match) return null; + + const highways = match[1].split('|'); + if ( + highways.length === 0 + || highways.some((highway) => !TRAFFIC_ROAD_HIGHWAYS.has(highway)) + ) return null; + + const south = Number(match[2]); + const west = Number(match[3]); + const north = Number(match[4]); + const east = Number(match[5]); + if ( + ![south, west, north, east].every(Number.isFinite) + || south >= north + || west >= east + || south < -90 || north > 90 + || west < -180 || east > 180 + || north - south > OVERPASS_MAX_BBOX_DEG + || east - west > OVERPASS_MAX_BBOX_DEG + ) return null; + + return { south, west, north, east, highways: new Set(highways) }; +} + +/** Decode the five XML entities that can occur in OSM attributes. */ +function decodeOsmXmlAttribute(value) { + return String(value || '') + .replaceAll('"', '"') + .replaceAll(''', "'") + .replaceAll('<', '<') + .replaceAll('>', '>') + .replaceAll('&', '&'); +} + +/** Read one standard double-quoted XML attribute from an element fragment. */ +function osmXmlAttribute(fragment, name) { + const match = String(fragment || '').match(new RegExp(`\\b${name}="([^"]*)"`)); + return match ? decodeOsmXmlAttribute(match[1]) : null; +} + +/** + * Convert the bounded OSM `/api/0.6/map` XML response into the Overpass-like + * `{elements:[{type:'way', tags, geometry}]}` shape consumed by traffic.js. + * The endpoint returns every referenced node alongside each way, so no extra + * node requests are necessary. + */ +export function parseOsmMapRoads(xml, highways = TRAFFIC_ROAD_HIGHWAYS) { + const nodes = new Map(); + const source = String(xml || ''); + + for (const match of source.matchAll(/]*?)\/>/g)) { + const id = osmXmlAttribute(match[1], 'id'); + const lat = Number(osmXmlAttribute(match[1], 'lat')); + const lon = Number(osmXmlAttribute(match[1], 'lon')); + if (id && Number.isFinite(lat) && Number.isFinite(lon)) { + nodes.set(id, { lat, lon }); + } + } + + const elements = []; + for (const match of source.matchAll(/]*)>([\s\S]*?)<\/way>/g)) { + const wayId = osmXmlAttribute(match[1], 'id'); + const body = match[2]; + const tags = {}; + for (const tagMatch of body.matchAll(/]*?)\/>/g)) { + const key = osmXmlAttribute(tagMatch[1], 'k'); + if (key) tags[key] = osmXmlAttribute(tagMatch[1], 'v') || ''; + } + if (!wayId || !highways.has(tags.highway)) continue; + + const geometry = []; + for (const ndMatch of body.matchAll(/]*?)\/>/g)) { + const node = nodes.get(osmXmlAttribute(ndMatch[1], 'ref')); + if (node) geometry.push(node); + } + if (geometry.length < 2) continue; + elements.push({ type: 'way', id: wayId, tags, geometry }); + } + + return { version: 0.6, generator: 'gods-eye-view-osm-map-fallback', elements }; +} + +/** + * Fetch traffic road geometry from the standard OSM map API. Returns null for + * non-traffic Overpass queries so the generic Overpass proxy remains unchanged. + */ +export async function fetchOsmMapRoadPayload(body, maxResponseBytes = OVERPASS_MAX_RESPONSE_BYTES, { + fetchImpl = fetch, + readBody = readResponseTextCapped, +} = {}) { + const spec = trafficRoadQuerySpec(body); + if (!spec) return null; + + const url = new URL(OSM_MAP_ENDPOINT); + url.searchParams.set('bbox', `${spec.west},${spec.south},${spec.east},${spec.north}`); + const controller = new AbortController(); + const timeoutId = setTimeout(() => controller.abort(), OSM_MAP_TIMEOUT_MS); + try { + const upstream = await fetchImpl(url, { + headers: { + Accept: 'application/xml', + 'User-Agent': 'gods-eye-view-osm-map/1.0 (local traffic layer)', + }, + signal: controller.signal, + }); + const responseBody = await readBody(upstream, maxResponseBytes); + if (!upstream.ok) throw new Error(`OSM map returned ${upstream.status}`); + if (!/ { // Only a 2xx is data. `< 500` cached every 4xx, so one mirror's // refusal was written to memory AND disk — and boundary-class