Merge aafa3064bb into 759652207f

CHANGELOG.md

@@ -7,6 +7,12 @@ of current runtime behavior, see [`docs/CURRENT-STATE.md`](docs/CURRENT-STATE.md
 
 ### Fixed
 
+- Street Traffic no longer remains in `SYNCING ROAD NETWORK` when the public
+  Overpass mirrors are unavailable. Exact bounded traffic road queries now use
+  OpenStreetMap's standard map endpoint first, translate its XML response into
+  the existing Overpass-shaped payload, and retain the generic Overpass mirror
+  rotation as a fallback. TomTom flow tiles and their attribution remain
+  unchanged.
 - Mapped-site outages show their scheduled retry countdown and distinguish
   known Overpass rate limits, timeouts, and query failures. Search feedback no
   longer claims a refresh succeeded while the layer is unavailable or loading.

DATA_SOURCES.md

@@ -24,7 +24,7 @@ How to read this:
 | **The Space Devs — Launch Library 2 v2.3** | Recent launch, payload, stage, and recovery metadata for Space Missions (30d) | [The Space Devs terms of use](https://github.com/TheSpaceDevs/Tutorials/blob/main/faqs/faq_TSD.md#terms-of-use): data may be used and shared in any form; avoid forwarding it without added value; attribution is encouraged (not mandatory). [Official API limits](https://ll.thespacedevs.com/docs/): 15 unauthenticated calls/hour; optional token | "Launch Library 2 — The Space Devs" (courtesy attribution) |
 | **Esri World Imagery** (ArcGIS Online tile service) | The keyless satellite basemap — the default landing when no Google/ion credential is configured, and the "Esri Satellite" map stack | [Esri Master Agreement](https://www.esri.com/en-us/legal/terms/full-master-agreement): the public World Imagery service is usable in public-facing apps with attribution; no key is required for this classic endpoint, but Esri governs and can change access — an app at scale should review current ArcGIS Location Platform terms | "Powered by Esri — Source: Esri, Maxar, Earthstar Geographics, and the GIS User Community" (provider carries the service's own credit line) |
 | **USGS** | Earthquakes | U.S. public domain | "Data courtesy of the U.S. Geological Survey" |
-| **OpenStreetMap (Overpass API)** | Road geometry for traffic | ODbL 1.0 | "© OpenStreetMap contributors" |
+| **OpenStreetMap (Standard Map API + Overpass API)** | Road geometry for traffic | ODbL 1.0 | "© OpenStreetMap contributors" |
 | **TomTom Traffic API** (flow vector tiles) | Live congestion coloring for the traffic layer (optional, BYOK) | [TomTom for Developers terms](https://developer.tomtom.com) (proprietary, your own key; free tier currently 200K tile requests/month — see [current pricing](https://docs.tomtom.com/pricing/)) | "Traffic flow data © TomTom" — registered when live mode activates |
 | **OpenStreetMap (Overpass API)** | Viewport-bounded mapped installation context for Global Context | ODbL 1.0 | "© OpenStreetMap contributors" (incomplete mapped context) |
 | **OpenStreetMap (Nominatim)** | Reverse-geocoded place label in the cockpit Local Info page | ODbL 1.0 + Nominatim usage policy | "© OpenStreetMap contributors" |
@@ -47,6 +47,7 @@ How to read this:
 - **TfL JamCams.** The camera list comes from the keyless `api.tfl.gov.uk` endpoint (an optional `TFL_APP_KEY` raises its rate limit); frames come from TfL's public S3 bucket. The "Powered by TfL Open Data" attribution is required by TfL's terms and is registered in the Data attribution popover.
 - **Radio Browser.** `/api/radio/stations` discovers official API mirrors, makes bounded and coalesced healthy/geolocated HTTPS-station queries, caches the normalized public-domain directory for 45 minutes, and may serve the last good catalog for up to seven days during an outage. Refreshes must meet minimum accepted-query and station coverage before replacing a warm catalog; schema-valid responses whose rows all fail the product's health policy do not count as successful queries. A usable partial cold catalog is explicitly `DEGRADED`, and malformed or empty successful payloads are rejected atomically. Every directory and click-count request rejects redirects, validates all resolved addresses as globally routable (including reserved/documentation IPv4 and special/non-global IPv6 exclusions), and pins the TLS connection to a validated address. Only MP3/AAC non-HLS directory rows with public HTTPS stream targets are returned; favicons are intentionally omitted. Pressing play connects one browser audio element directly to the selected broadcaster and calls the directory's click counter through known-ID-only `POST /api/radio/click/:uuid`. GEV never proxies, caches, records, bundles, or redistributes audio. Radio Browser supplies station-level tags, not dependable current-song or upcoming-program metadata, so Radio filtering never claims either. Direct playback exposes the listener's IP address to the broadcaster, whose own stream terms apply.
 - **TomTom Traffic.** Optional and BYOK: without `TOMTOM_API_KEY` the traffic layer runs its built-in simulation and no TomTom data (or attribution) appears. With a key, flow vector tiles are fetched through the server-side `/api/tomtom` proxy (120 s cache + a daily tile-budget governor — `TOMTOM_DAILY_TILE_BUDGET`, default 40,000, a configurable application safety ceiling, not a guarantee of staying within TomTom's monthly free allowance; TomTom's [current pricing](https://docs.tomtom.com/pricing/) lists 200K free tile requests per month) and the "Traffic flow data © TomTom" credit is registered in the Data attribution popover the moment live mode activates. TomTom data is served live and cached only transiently (≤120 s TTL under `.gev-cache/`, gitignored) — it is not bundled or redistributed. One 23 KB point-in-time tile snapshot is committed as a decode-test fixture (`src/data/fixtures/`, © TomTom, never served to the app).
+- **OpenStreetMap road geometry.** Street Traffic's exact, bounded highway query uses the standard `/api/0.6/map` endpoint first, parses its XML response server-side into the existing Overpass-shaped road payload, and keeps the established four-mirror Overpass rotation as a fallback. Other Overpass-backed features continue to use the generic query path. Road geometry is fetched at runtime, is not bundled or redistributed, and remains covered by the in-app "© OpenStreetMap contributors" attribution.
 - **Re:Earth Terrain.** Keyless (no API key). Used two ways: (1) `src/mapStackController.js` swaps in a `Cesium.CesiumTerrainProvider` pointed at Re:Earth's `cesium-mesh/ellipsoid` quantized-mesh endpoint for globe stacks without a Cesium ion token (e.g. OSM), replacing a flat `EllipsoidTerrainProvider`; falls back to the flat provider if the endpoint can't be reached. (2) The server-side `/api/terrain/heights` proxy (disk-cached, serve-stale) resolves per-point ellipsoidal ground height for entity placement. Both are best-effort with a keyless-safe fallback (bundled EGM96 geoid math) if Re:Earth is unreachable.
 - **Global Context installation context.** `/api/military-installations` queries only an allow-listed subset of OSM `military=*` and `landuse=military` features inside a maximum 10° non-dateline viewport. It caches and may serve stale mapped context, but it is neither a global installation database nor evidence of capability, activity, or absence. User-requested Google Places results remain separately sourced candidates unless their returned types explicitly establish military classification; generic offices, museums, and similarly ambiguous matches are excluded from military proximity counts.
 - **Cockpit regional briefing.** `/api/regional-brief` rounds aircraft coordinates into 0.1° cache cells, caches results for five minutes, and serializes Nominatim calls at no more than one request per second. Google News RSS is queried with the resolved locality/region first; GDELT is used only when that RSS query fails or is empty. Google's published Google News terms restrict that source to personal, noncommercial use, so commercial deployments must disable/replace it or obtain separate permission; GDELT permits commercial dataset use with citation. The Data attribution popover identifies the active headline sources; article links retain publisher attribution. Headlines are location-query matches, not verified incidents, risk rankings, or evidence that a location is safe. Empty, partial, stale, and unavailable source states remain distinct. Open-Meteo supplies current conditions independently of the news source. `WX OFF` disables cockpit weather rendering only; the Local Info briefing still fetches its source-backed weather values and displays the required linked Open-Meteo credit.

docs/CURRENT-STATE.md

@@ -2030,6 +2030,7 @@ silently demoting every later lookup for the session.
 - `/api/overpass` fans out across four public mirrors. `overpassPayloadIsData()` governs cache reads, writes, and stale fallback: only a 2xx that is neither rate-limited nor a body-level runtime error qualifies. Previously stored refusals are ignored on both fresh and stale reads, so upgrading does not require manually clearing the disk cache.
 - HTTP refusals such as 406 now rotate alongside the existing network, rate-limit, and runtime-error cases. A refusal from one mirror no longer prevents reaching healthy alternatives or persists under the seven-day road/month-long boundary cache TTLs. Concurrent identical queries share one mirror sequence; if it fails, both the initiating and joined callers can use the same last-good data.
 - A refusal every mirror agrees on is still reported with the first mirror's status and body, so a genuinely malformed query says what upstream said — but only after every mirror has had the chance to answer it. `fetchOverpassPayload` takes injectable endpoints and fetch so the rotation is tested without a live mirror (`src/overpassProxy.test.mjs`).
+- Street Traffic's exact bounded highway queries take a separate 12-second path through OpenStreetMap's standard `/api/0.6/map` endpoint. The server parses its XML nodes, ways, and tags into the existing Overpass-shaped road payload, so `traffic.js` and TomTom matching do not need a second client contract. If the OSM map request fails — or the query is not the traffic shape — the established four-mirror Overpass path remains the fallback; generic Overpass-backed features are unchanged.
 
 ### Share-link v2 layer state (August 2026)
 
@@ -2543,10 +2544,11 @@ easier to meet (detection is now on more often), but does not create it.
   is configured (env or Keychain `tomtom-api`/`api-key`), which enables `live` mode:
   TomTom flow vector tiles via the budget-governed `/api/tomtom` proxy
   (`.gev-cache/tomtom/`, 120 s TTL, `TOMTOM_DAILY_TILE_BUDGET` default 40k/day),
-  decoded client-side (`flowTiles.js`), matched onto Overpass roads
+  decoded client-side (`flowTiles.js`), matched onto OSM roads
   (`flowMatch.js`), and rendered as green/amber/red dot color + speed/density
   scaling (`trafficFlowStyle.js`); closures spawn no dots; unmatched roads stay
-  white. Road fetch bounds center on the camera look-at point (`trafficBounds.js`).
+  white. Road fetch bounds center on the camera look-at point (`trafficBounds.js`);
+  the proxy uses the bounded OSM map path before falling back to Overpass.
 - Development captures opened with `?trafficDebug=1` mint an interaction anchor
   from the exact `camera.changed` event that arms each debounced load, then emit
   scheduling-correlated User Timing entries for production `response.json`, road

src/data/traffic.js

@@ -18,15 +18,17 @@ import { holdContinuousRender, releaseContinuousRender } from '../renderGovernor
  * @file Street Traffic — animated dots along OSM road polylines, colored by
  * live TomTom congestion when a key is configured.
  *
- * Road geometry: OSM Overpass API (free, no auth). Fetches road polylines for
+ * Road geometry: OpenStreetMap's standard map endpoint for bounded traffic
- * the camera viewport, spawns PointPrimitives that lerp along pre-computed
+ * queries, with the server-side Overpass proxy as fallback. Fetches road
- * Cartesian3 waypoints. Camera-gated: only active below ~8 km altitude.
+ * polylines for the camera viewport, spawns PointPrimitives that lerp along
+ * pre-computed Cartesian3 waypoints. Camera-gated: only active below ~8 km
+ * altitude.
  *
  * Two modes (decided once per session via `/api/tomtom/status`):
  *  - `sim` (keyless default): white dots at hardcoded per-road-class speeds —
  *    the original simulation, byte-identical behavior.
  *  - `live`: TomTom flow tiles (`flowTiles.js`) are matched onto the same
- *    Overpass roads (`flowMatch.js`); matched roads color/slow/densify their
+ *    OSM roads (`flowMatch.js`); matched roads color/slow/densify their
  *    dots by real congestion (`trafficFlowStyle.js`), closed roads spawn no
  *    dots, and unmatched roads keep the simulated white.
  *

src/overpassProxy.test.mjs

@@ -11,7 +11,13 @@ import { mkdir, readFile, writeFile, unlink } from 'node:fs/promises';
 import { createHash, randomUUID } from 'node:crypto';
 import path from 'node:path';
 import { Readable } from 'node:stream';
-import createViteConfig, { fetchOverpassPayload, overpassPayloadIsData, readOverpassDisk } from '../vite.config.js';
+import createViteConfig, {
+  fetchOsmMapRoadPayload,
+  fetchOverpassPayload,
+  overpassPayloadIsData,
+  parseOsmMapRoads,
+  readOverpassDisk,
+} from '../vite.config.js';
 
 const ENDPOINTS = ['https://a.example/api', 'https://b.example/api', 'https://c.example/api'];
 
@@ -39,6 +45,49 @@ const run = (byUrl) => {
 
 const DATA = { status: 200, body: '{"elements":[]}' };
 
+const OSM_ROAD_XML = `<?xml version="1.0" encoding="UTF-8"?>
+<osm version="0.6" generator="test">
+  <node id="1" lat="-3.7600" lon="-38.4900" />
+  <node id="2" lat="-3.7590" lon="-38.4890" />
+  <node id="3" lat="-3.7580" lon="-38.4880" />
+  <way id="101">
+    <nd ref="1"/><nd ref="2"/><nd ref="3"/>
+    <tag k="highway" v="primary"/><tag k="oneway" v="yes"/>
+  </way>
+  <way id="102">
+    <nd ref="1"/><nd ref="3"/>
+    <tag k="highway" v="residential"/>
+  </way>
+</osm>`;
+
+test('OSM map XML converts referenced nodes into traffic road geometry', () => {
+  const payload = parseOsmMapRoads(OSM_ROAD_XML, new Set(['primary']));
+  assert.equal(payload.elements.length, 1);
+  assert.equal(payload.elements[0].id, '101');
+  assert.equal(payload.elements[0].tags.highway, 'primary');
+  assert.equal(payload.elements[0].tags.oneway, 'yes');
+  assert.deepEqual(payload.elements[0].geometry, [
+    { lat: -3.76, lon: -38.49 },
+    { lat: -3.759, lon: -38.489 },
+    { lat: -3.758, lon: -38.488 },
+  ]);
+});
+
+test('OSM map road fallback requests the traffic bbox and returns Overpass-shaped JSON', async () => {
+  let requestedUrl = null;
+  const query = '[out:json][timeout:12];(way["highway"~"^(motorway|trunk|primary|secondary)$"](-3.7609,-38.4967,-3.7540,-38.4827););out geom qt;';
+  const payload = await fetchOsmMapRoadPayload(`data=${encodeURIComponent(query)}`, 1e6, {
+    fetchImpl: async (url) => {
+      requestedUrl = String(url);
+      return new Response(OSM_ROAD_XML, { status: 200, headers: { 'content-type': 'application/xml' } });
+    },
+  });
+  assert.equal(payload.status, 200);
+  assert.equal(payload.endpoint, 'https://api.openstreetmap.org/api/0.6/map');
+  assert.match(requestedUrl, /bbox=-38\.4967%2C-3\.7609%2C-38\.4827%2C-3\.754/);
+  assert.equal(JSON.parse(payload.body).elements.length, 1);
+});
+
 test('disk cache rejects old refusals for fresh and stale reads but preserves last-good data', async () => {
   const key = `overpass-cache-regression-${randomUUID()}`;
   const directory = path.join(process.cwd(), '.gev-cache', 'overpass');

vite.config.js

@@ -198,6 +198,8 @@ const OVERPASS_UPSTREAMS = [
   // Verified: planet coverage (Texas query), CORS *, ~5-20 s cold latency.
   'https://overpass.private.coffee/api/interpreter',
 ];
+/** Standard OSM map endpoint used as a bounded road-geometry fallback. */
+const OSM_MAP_ENDPOINT = 'https://api.openstreetmap.org/api/0.6/map';
 /**
  * TTL for FRESH cached Overpass responses (ms). Road geometry is static for
  * months — the original 45 s TTL forced a public-mirror round-trip on nearly
@@ -221,6 +223,8 @@ const OVERPASS_BOUNDARY_DISK_TTL_MS = 30 * 86_400_000;
 const OVERPASS_DISK_DIR = path.join(process.cwd(), '.gev-cache', 'overpass');
 /** Per-upstream fetch timeout (ms). */
 const OVERPASS_TIMEOUT_MS = 22000;
+/** OSM map fallback timeout (ms). */
+const OSM_MAP_TIMEOUT_MS = 12000;
 /** Max entries in the Overpass response cache (LRU-like, oldest evicted first). */
 const OVERPASS_CACHE_MAX_ENTRIES = 120;
 /** @type {Map<string,{status:number,body:string,contentType:string,endpoint:string,cachedAt:number}>} */
@@ -2573,6 +2577,165 @@ export function overpassPayloadIsData(payload) {
     && !payload.runtimeError;
 }
 
+const TRAFFIC_ROAD_HIGHWAYS = new Set([
+  'motorway', 'trunk', 'primary', 'secondary',
+  'tertiary', 'residential', 'unclassified',
+]);
+
+/**
+ * Extract the bounded road query shape emitted by src/data/traffic.js. The
+ * standard OSM map endpoint cannot execute arbitrary Overpass QL, so the
+ * fallback is deliberately limited to this exact highway+bbox form.
+ */
+function trafficRoadQuerySpec(body) {
+  let query;
+  try {
+    query = new URLSearchParams(String(body || '')).get('data');
+  } catch {
+    return null;
+  }
+  if (!query) return null;
+
+  const match = query.match(
+    /way\s*\[\s*"highway"\s*~\s*"\^\(([^\"]+)\)\$"\s*\]\s*\(\s*(-?\d+(?:\.\d+)?)\s*,\s*(-?\d+(?:\.\d+)?)\s*,\s*(-?\d+(?:\.\d+)?)\s*,\s*(-?\d+(?:\.\d+)?)\s*\)\s*;/i,
+  );
+  if (!match) return null;
+
+  const highways = match[1].split('|');
+  if (
+    highways.length === 0
+    || highways.some((highway) => !TRAFFIC_ROAD_HIGHWAYS.has(highway))
+  ) return null;
+
+  const south = Number(match[2]);
+  const west = Number(match[3]);
+  const north = Number(match[4]);
+  const east = Number(match[5]);
+  if (
+    ![south, west, north, east].every(Number.isFinite)
+    || south >= north
+    || west >= east
+    || south < -90 || north > 90
+    || west < -180 || east > 180
+    || north - south > OVERPASS_MAX_BBOX_DEG
+    || east - west > OVERPASS_MAX_BBOX_DEG
+  ) return null;
+
+  return { south, west, north, east, highways: new Set(highways) };
+}
+
+/** Decode the five XML entities that can occur in OSM attributes. */
+function decodeOsmXmlAttribute(value) {
+  return String(value || '')
+    .replaceAll('&quot;', '"')
+    .replaceAll('&apos;', "'")
+    .replaceAll('&lt;', '<')
+    .replaceAll('&gt;', '>')
+    .replaceAll('&amp;', '&');
+}
+
+/** Read one standard double-quoted XML attribute from an element fragment. */
+function osmXmlAttribute(fragment, name) {
+  const match = String(fragment || '').match(new RegExp(`\\b${name}="([^"]*)"`));
+  return match ? decodeOsmXmlAttribute(match[1]) : null;
+}
+
+/**
+ * Convert the bounded OSM `/api/0.6/map` XML response into the Overpass-like
+ * `{elements:[{type:'way', tags, geometry}]}` shape consumed by traffic.js.
+ * The endpoint returns every referenced node alongside each way, so no extra
+ * node requests are necessary.
+ */
+export function parseOsmMapRoads(xml, highways = TRAFFIC_ROAD_HIGHWAYS) {
+  const nodes = new Map();
+  const source = String(xml || '');
+
+  for (const match of source.matchAll(/<node\b([^>]*?)\/>/g)) {
+    const id = osmXmlAttribute(match[1], 'id');
+    const lat = Number(osmXmlAttribute(match[1], 'lat'));
+    const lon = Number(osmXmlAttribute(match[1], 'lon'));
+    if (id && Number.isFinite(lat) && Number.isFinite(lon)) {
+      nodes.set(id, { lat, lon });
+    }
+  }
+
+  const elements = [];
+  for (const match of source.matchAll(/<way\b([^>]*)>([\s\S]*?)<\/way>/g)) {
+    const wayId = osmXmlAttribute(match[1], 'id');
+    const body = match[2];
+    const tags = {};
+    for (const tagMatch of body.matchAll(/<tag\b([^>]*?)\/>/g)) {
+      const key = osmXmlAttribute(tagMatch[1], 'k');
+      if (key) tags[key] = osmXmlAttribute(tagMatch[1], 'v') || '';
+    }
+    if (!wayId || !highways.has(tags.highway)) continue;
+
+    const geometry = [];
+    for (const ndMatch of body.matchAll(/<nd\b([^>]*?)\/>/g)) {
+      const node = nodes.get(osmXmlAttribute(ndMatch[1], 'ref'));
+      if (node) geometry.push(node);
+    }
+    if (geometry.length < 2) continue;
+    elements.push({ type: 'way', id: wayId, tags, geometry });
+  }
+
+  return { version: 0.6, generator: 'gods-eye-view-osm-map-fallback', elements };
+}
+
+/**
+ * Fetch traffic road geometry from the standard OSM map API. Returns null for
+ * non-traffic Overpass queries so the generic Overpass proxy remains unchanged.
+ */
+export async function fetchOsmMapRoadPayload(body, maxResponseBytes = OVERPASS_MAX_RESPONSE_BYTES, {
+  fetchImpl = fetch,
+  readBody = readResponseTextCapped,
+} = {}) {
+  const spec = trafficRoadQuerySpec(body);
+  if (!spec) return null;
+
+  const url = new URL(OSM_MAP_ENDPOINT);
+  url.searchParams.set('bbox', `${spec.west},${spec.south},${spec.east},${spec.north}`);
+  const controller = new AbortController();
+  const timeoutId = setTimeout(() => controller.abort(), OSM_MAP_TIMEOUT_MS);
+  try {
+    const upstream = await fetchImpl(url, {
+      headers: {
+        Accept: 'application/xml',
+        'User-Agent': 'gods-eye-view-osm-map/1.0 (local traffic layer)',
+      },
+      signal: controller.signal,
+    });
+    const responseBody = await readBody(upstream, maxResponseBytes);
+    if (!upstream.ok) throw new Error(`OSM map returned ${upstream.status}`);
+    if (!/<osm\b/i.test(responseBody)) throw new Error('OSM map returned invalid XML');
+    const payload = parseOsmMapRoads(responseBody, spec.highways);
+    return {
+      status: 200,
+      body: JSON.stringify(payload),
+      contentType: 'application/json',
+      endpoint: OSM_MAP_ENDPOINT,
+      rateLimited: false,
+      runtimeError: false,
+    };
+  } finally {
+    clearTimeout(timeoutId);
+  }
+}
+
+/** Traffic gets a fast bounded OSM map path, with generic Overpass as fallback. */
+async function fetchTrafficRoadGeometry(body) {
+  try {
+    const payload = await fetchOsmMapRoadPayload(body);
+    if (payload) {
+      console.log('[Overpass Proxy] traffic roads via OSM map API');
+      return payload;
+    }
+  } catch (error) {
+    console.warn('[Overpass Proxy] OSM map traffic path failed:', error?.message || error);
+  }
+  return fetchOverpassPayload(body);
+}
+
 /**
  * Try each mirror once, retaining response-size and per-mirror timeout caps.
  * Refusals and body-level failures rotate; total failure returns the last
@@ -2757,7 +2920,7 @@ function overpassProxy() {
             return;
           }
           _overpassConcurrent += 1;
-          const requestPromise = fetchOverpassPayload(safeBody)
+          const requestPromise = fetchTrafficRoadGeometry(safeBody)
             .then((payload) => {
               // Only a 2xx is data. `< 500` cached every 4xx, so one mirror's
               // refusal was written to memory AND disk — and boundary-class