diff --git a/AGENTS.md b/AGENTS.md index 3f8581b808..4547b149c4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -91,6 +91,32 @@ When you are creating a plan file in the repository itself, new plan documents b 6. Attach inspectable generated artifacts. When your task produces a user-inspectable deliverable file, follow the Paperclip skill's "Generated Artifacts and Work Products" workflow before final disposition. In this repo, prefer the self-contained skill helper at `skills/paperclip/scripts/paperclip-upload-artifact.sh` so the file is available through the Paperclip API, create/update an artifact work product when the file is the deliverable, link the uploaded artifact in the final issue comment, and then set status. Do not rely on local filesystem paths as the only access path. If an important file intentionally remains workspace-only, create/update a work product with `metadata.resourceRef.kind: "workspace_file"` and a workspace-relative path, then name that work product and path in the final comment. Treat browse/search as a fallback for recovering workspace files, not the preferred deliverable path. See `doc/AGENT-ARTIFACTS.md` for details and `.mp4`/`.webm` examples. +7. Name the three data paths correctly. +This repo has three separate data paths. Do not confuse them. Match a change to a path by its file path, not by the word "observability" or "telemetry" alone. + +- **Telemetry** is the Paperclip first-party event system. It is opt-out and it sends data to a Paperclip endpoint by default. Its paths are: + - `packages/shared/src/telemetry/` + - the generated contract `packages/shared/src/telemetry/generated/paperclip-telemetry.ts` + - each caller of `packages/shared/src/telemetry/events.ts` or `packages/shared/src/telemetry/client.ts` +- **Observability** is the OpenTelemetry trace path. An operator must set an OTLP endpoint. Until an operator sets the endpoint, the tracer is a no-operation. Its paths are: + - `server/src/instrumentation.ts` + - `doc/observability.md` + - `packages/adapter-utils/src/duplex-observability.ts` + - `server/src/services/duplex-observability-recorder.ts` + - the span attributes in `packages/adapter-utils/src/acpx-engine/startup-timing.ts` +- **The run log** holds rows in the local `heartbeat_run_events` table. The data stays in the instance database. Its paths are: + - `doc/run-log-events.md` + - `packages/db/src/schema/heartbeat_run_events.ts` + - the append path `appendRunEvent` in `server/src/services/heartbeat.ts` + +Apply a review level that matches the path: + +- **Telemetry change (strict review).** The author updates the generated contract first. The author updates `packages/shared/src/telemetry/README.md` in the same pull request. The author requests a privacy review. Reason: a Telemetry event goes to a Paperclip endpoint by default, so a mistake sends data immediately. +- **Observability change (lighter review).** The operator endpoint gate stays in place. The no-operation behaviour stays when no endpoint is set. A privacy review is not necessary while the change stays inside the closed span-attribute allowlist. +- **Run-log change (no extra review).** A run-log change needs neither review level above, because the data stays in the instance database. + +**Exclusion.** The word "observability" in a file such as `server/src/services/recovery-observability.ts` names a different concept. Apply this rule by path, not by word match. + ## 6. Database Change Workflow When changing data model: diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 152c527f20..a1dab6f5f3 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -104,7 +104,9 @@ All tests must pass before a PR can be merged. Run them locally first and verify ### Telemetry Changes -If your change adds, removes, or modifies emitted telemetry events, update the [Telemetry Data Contract](packages/shared/src/telemetry/README.md) in the same PR. Keep clients emitting raw dimension values and avoid documenting or relying on private delivery details. +This repo has three separate data paths: Telemetry, Observability, and the run log. See rule 7 in `AGENTS.md` for the full definitions and the review level each path needs. + +If your change adds, removes, or modifies emitted telemetry events, update the [Telemetry Data Contract](packages/shared/src/telemetry/README.md) in the same PR. Keep clients emitting raw dimension values and avoid documenting or relying on private delivery details. If your change adds, removes, or modifies an OpenTelemetry span or span attribute, keep the change inside the closed span-attribute allowlist in `packages/adapter-utils/src/acpx-engine/startup-timing.ts`. If your change adds or modifies a run-log event, update `doc/run-log-events.md` in the same PR. ### Paperclip Gates Must Pass diff --git a/doc/acp-run-lifecycle.md b/doc/acp-run-lifecycle.md index ea68b76d1c..e4d0c566d0 100644 --- a/doc/acp-run-lifecycle.md +++ b/doc/acp-run-lifecycle.md @@ -80,13 +80,15 @@ second run blocked on the lease until the first run fully returns, so the second run never re-stages into a workspace the first run still uses. The release runs in a `finally`, so an earlier teardown fault never strands the lease. -## Per-phase telemetry +## Per-phase run-log events -The run emits one telemetry event per named lifecycle phase. Each event carries -only the phase name, the wall-time duration, and the outcome (`ok` or `failed`). -The phase name is one member of a closed allowlist. An event never carries a +The run writes one [run-log event](run-log-events.md) per named lifecycle +phase, to the `heartbeat_run_events` table. This event is not a Paperclip +Telemetry event and not an OpenTelemetry export. Each event carries only the +phase name, the wall-time duration, and the outcome (`ok` or `failed`). The +phase name is one member of a closed allowlist. An event never carries a command, an argument, a path, an environment value, or a raw identifier. A -telemetry failure never fails the run. +run-log write failure never fails the run. ## Known limitations and deferred work @@ -97,5 +99,5 @@ telemetry failure never fails the run. run-scoped credential rebind protocol exists. - **The sandbox staged-files reuse stays enabled.** Its reuse payload carries no credential, so a compatible resume reuses the already-staged runtime. -- **The per-phase telemetry is observability-only.** It records the duration and - the outcome of each phase; it does not change run control flow. +- **The per-phase run-log events record the duration and the outcome only.** + They never change run control flow. diff --git a/doc/observability.md b/doc/observability.md index 15101f47d9..5b86a4e9f7 100644 --- a/doc/observability.md +++ b/doc/observability.md @@ -1,5 +1,10 @@ # Observability +This document is the Observability contract. It covers the OpenTelemetry +trace path and two local instrumentation contracts; see the +[Telemetry Data Contract](../packages/shared/src/telemetry/README.md) for the +separate first-party event system. + Paperclip ships with **opt-in** OpenTelemetry auto-instrumentation for the server process. When activated it produces **traces only** — no metrics and no logs are exported by this integration. The OTel packages are *optional peer @@ -94,8 +99,332 @@ without tracing — your server stays up. ## Scope -This integration emits **traces only**. Metrics and log exporters are out of -scope and intentionally not configured here. Auto-instrumentations for -`fs`, `dns`, and `net` are disabled by default because they are too chatty +The OpenTelemetry export carries **traces only**. Metrics and log exporters +are out of scope and intentionally not configured here. Auto-instrumentations +for `fs`, `dns`, and `net` are disabled by default because they are too chatty for this workload; everything else from `@opentelemetry/auto-instrumentations-node` is on (HTTP, Express, PG, etc.). + +This document also holds two local instrumentation contracts: the sandbox +startup trace spans, and the sandbox duplex transport instrumentation. Both +sections follow below. + +## Sandbox Startup Trace Spans + +Paperclip opens OpenTelemetry spans on the sandbox start path. These spans are +an Observability surface. They are not Paperclip Telemetry events. The +generated telemetry contract does not cover them, so this section is their +canonical contract. + +The spans are opt-in. Paperclip exports them only when an OTLP endpoint is +configured. With no endpoint the whole span path is a no-op. Paperclip opens the +spans only for a run that targets a remote sandbox. A local run and an SSH run +stay out of these spans. + +Every span attribute uses the closed `paperclip.sandbox.startup.` prefix and +rides a fixed allowlist. A command line, an argument, an environment value, a +file path, program output, or a raw identifier never rides a span. It rides +neither as an attribute nor as an event. The producer bounds each free-form +value: + +- A command basename maps to a small known set. Any other value maps to `other`. +- A region maps to a small known set. Any other value maps to `unknown`. +- An image id, a sandbox id, and a lease id ride only as a non-reversible short + hash. + +Each numeric attribute is finite. Paperclip omits an attribute when its value is +absent, never a misleading `0`. + +### Spans + +| Span | Scope | Parent | +| --- | --- | --- | +| `sandbox.startup` | The one root span for a sandbox bring-up. | none (root) | +| `workspace.resolve` | Workspace resolution step. | `sandbox.startup` | +| `codex-home.seed` | Managed-home seed step. | `sandbox.startup` | +| `skills.reconcile` | Skills reconcile step. | `sandbox.startup` | +| `stage.sync` | Workspace stage-sync step. | `sandbox.startup` | +| `snapshot.git` | Host-side git workspace enumeration inside `stage.sync` (`git status --ignored`, the HEAD diffs, `ls-files`). | `stage.sync` | +| `snapshot.baseline` | Host-side baseline workspace content-hash walk inside `stage.sync`, kept for restore. | `stage.sync` | +| `stage.workspace` | One inbound workspace stage task inside `stage.sync`. It packs and uploads the workspace. | `stage.sync` | +| `stage.asset.` | One inbound asset stage task inside `stage.sync`. It packs and uploads one managed-home asset. The `` segment is the asset key. | `stage.sync` | +| `stage.project.` | One inbound referenced-project stage task inside `stage.sync`. It uploads one referenced project. The `` segment is the project id. | `stage.sync` | +| `pack` | Host-side workspace tarball build inside the `stage.workspace` task. | `stage.workspace` | +| `bridge.paperclip` | Paperclip bridge start step. | `sandbox.startup` | +| `bridge.process-session` | Process-session bridge start step. | `sandbox.startup` | +| `acp.handshake` | ACP session handshake step. | `sandbox.startup` | +| `sandbox.syncBack` | The settlement sync-back that restores the managed home at teardown. | the active run span | +| `restore.workspace` | One outbound workspace restore task at teardown. It reads the sandbox workspace back and merges it into the host workspace. | `sandbox.syncBack` | +| `restore.asset.` | One outbound asset restore task at teardown. It reads one asset back to its host store. The `` segment is the asset key. | `sandbox.syncBack` | +| `sandbox.agentSession.sendInput` | One outbound ACP message to the agent — the socket handler's one `writeTextFile` exec. | the active run span | +| `sandbox.agentSession.pollOutput` | One 100 ms poll tick — `list`, then `read`+`remove` per file found (`1 + 2n` execs). | the active run span | +| `sandbox.callbackBridge.relayRequest` | One Paperclip-API callback request — read the request, write the response, remove it. | the active run span | +| `sandbox.agentProcess` | The persistent streamed agent process the process-session bridge launches; open until the process settles or the bridge tears down, whichever comes first. | the active run span | +| `sandbox.exec` | One host-to-sandbox execution. | the active step or wrapper span | + +A step span name is the step name. The `sandbox.exec` span parents to the step +span that runs the execution, so each execution nests under its step. Within +`stage.sync`, the host-side sub-steps `snapshot.git` and `snapshot.baseline` open +as child spans of the step, so the host work at the head of the step is +attributed rather than showing as a gap. Each inbound sync operation also opens +its own task span under `stage.sync`: `stage.workspace`, one `stage.asset.` +per asset, and one `stage.project.` per referenced project. The `pack` span +nests under `stage.workspace`, because the host builds the tarball inside that +task. Two concurrent tasks produce overlapping spans. + +The settlement `sandbox.syncBack` span runs at teardown and parents to the run +span. It wraps the managed-home restore. Each outbound restore operation opens +its own task span under `sandbox.syncBack`: `restore.workspace` and one +`restore.asset.` per asset. Two concurrent restore tasks produce overlapping +spans. A run-time +`sandbox.exec` span parents instead to the run-time wrapper span that runs it +(`sandbox.agentSession.sendInput`, `sandbox.agentSession.pollOutput`, +`sandbox.callbackBridge.relayRequest`, or `sandbox.agentProcess`). Each run-time +wrapper span parents to the live run span (`agent.turn` during the turn, +`task.run` otherwise). With no active trace context the exec span opens +unparented. + +`sandbox.agentProcess` wraps the persistent streamed agent process. The +process-session bridge launches it during `bridge.process-session`, so it opens +under `task.run` — no turn has started yet. It therefore overlaps the sibling +`agent.turn` rather than nesting under it or dangling off the short-lived bring-up +step. The span ends when the process settles or when the bridge tears down, +whichever comes first. The bridge tears down before the run root span ends, so +the span never outlives `task.run` even when the process lingers past teardown +(the sandbox `execute` has no cancel, so a lingering process cannot be forced to +resolve). + +The root span sets the error status when the bring-up fails. Each step span sets +the error status when its step fails. The `sandbox.exec` span sets the error +status when the exit code is non-zero or the execution throws. + +### Outcome values + +The `paperclip.sandbox.startup.outcome` attribute uses a closed value set: + +- `ok` — the step or the execution settled with a success result. +- `skipped` — a warm cache skipped the step; the step ran no work. +- `failed` — the step or the execution threw, or the exit code was non-zero. + +### Root span attributes + +The `sandbox.startup` root span uses this closed attribute allowlist. + +| Attribute | Type | Optional | Meaning | +| --- | --- | --- | --- | +| `paperclip.sandbox.startup.root.wall_ms` | number | no | The root-span wall time of the whole bring-up. | +| `paperclip.sandbox.startup.root.work_ms` | number | no | The sum of the step wall times. | +| `paperclip.sandbox.startup.root.diff_ms` | number | no | `work_ms − wall_ms`; the overlap the parallel steps saved. | +| `paperclip.sandbox.startup.provider` | string | yes | The normalized provider family. | +| `paperclip.sandbox.startup.cold_start` | boolean | yes | Whether the bring-up is a cold start. | +| `paperclip.sandbox.startup.region` | string | yes | The clamped region label. | +| `paperclip.sandbox.startup.image_id` | string | yes | The hashed image id. | +| `paperclip.sandbox.startup.sandbox_id` | string | yes | The hashed sandbox id. | +| `paperclip.sandbox.startup.lease_id` | string | yes | The hashed lease id. | + +### Step span attributes + +Each bring-up step span uses this closed attribute allowlist. The step name +rides the span name, so no `step` attribute repeats it. + +| Attribute | Type | Optional | Meaning | +| --- | --- | --- | --- | +| `paperclip.sandbox.startup.step.wall_ms` | number | no | The wall time of the step. | +| `paperclip.sandbox.startup.outcome` | string | no | The step outcome (`ok`, `skipped`, or `failed`). | +| `paperclip.sandbox.startup.provider` | string | yes | The normalized provider family. | +| `paperclip.sandbox.startup.batch` | string | yes | A shared tag that marks two parallel steps as one batch. | +| `paperclip.sandbox.startup.handshake.create_runtime.wall_ms` | number | yes | The create-runtime sub-time of the `acp.handshake` step. | +| `paperclip.sandbox.startup.handshake.ensure_session.wall_ms` | number | yes | The ensure-session sub-time of the `acp.handshake` step. | + +The round-trip count and the provider durations no longer ride a step span. The +per-execution `sandbox.exec` child spans carry that detail. + +### `sandbox.exec` span attributes + +The `sandbox.exec` span uses this closed attribute allowlist. Paperclip omits a +numeric attribute when the provider does not report the value. + +| Attribute | Type | Optional | Meaning | +| --- | --- | --- | --- | +| `paperclip.sandbox.startup.provider` | string | no | The normalized provider family. | +| `paperclip.sandbox.startup.exec.command` | string | no | The clamped `argv[0]` command label. | +| `paperclip.sandbox.startup.exec.exit_code` | number | yes | The numeric process exit code. | +| `paperclip.sandbox.startup.exec.wall_ms` | number | no | The host-measured wall time of the execution. | +| `paperclip.sandbox.startup.exec.wait_before_ms` | number | yes | The provider handle-fetch wait before the execution ran. | +| `paperclip.sandbox.startup.exec.sandbox_ms` | number | yes | The in-sandbox run time of the execution. | +| `paperclip.sandbox.startup.exec.network_ms` | number | yes | The transport time the host adds; `wall_ms − wait_before_ms − sandbox_ms`. | +| `paperclip.sandbox.startup.exec.critical_path` | boolean | no | Whether the execution sits on the startup critical path. | +| `paperclip.sandbox.startup.exec.cache_hit` | boolean | yes | Whether the provider served the sandbox handle from its warm cache. | +| `paperclip.sandbox.startup.outcome` | string | no | The execution outcome (`ok` or `failed`). | + +The plugin decides the cache hit at the sandbox-handle lookup. The span no +longer infers a cache hit from `wait_before_ms == 0`. Paperclip omits the +`cache_hit` attribute when the provider does not report the value. + +To add a span attribute, extend the `SANDBOX_STARTUP_SPAN_ATTRS` allowlist in +the code first. Keep the attribute low-cardinality and free of user content. + +### Provider spans + +A sandbox provider plugin also opens spans for its own sync steps. These spans +use the `sandbox.daytona.` name prefix. They share the +`paperclip.sandbox.startup.` attribute prefix and obey the same opt-in and +no-user-content rules as the startup spans above. + +The plugin worker runs in a separate process from the host. So the host treats +every field of a worker-sent span as untrusted input. The host re-clamps the +span name and every attribute at one boundary, the `span.record` host handler, +before it records the span. + +| Span | Scope | Parent | +| --- | --- | --- | +| `sandbox.daytona.pack` | The host-local pack step that builds the upload tarball. It makes no sandbox round trip. | the active startup step span | +| `sandbox.daytona.transfer` | The transfer step: an upload to the sandbox (inbound) or a download from the sandbox (outbound). The `paperclip.sandbox.startup.transfer.direction` attribute records the direction. | the active sync task span (`stage.*` inbound, `restore.*` under `sandbox.syncBack` outbound) | +| `sandbox.daytona.ensureDirectory` | The `mkdir -p` step that ensures a directory exists before a write. | the active startup step span | +| `sandbox.daytona.checkSymlinkEscape` | The re-check step that a path resolves inside the workspace root before use. | the active startup step span | +| `sandbox.daytona.promote` | The atomic move of a staged temp onto its target via a pinned dir handle. | the active startup step span | +| `sandbox.daytona.extractTarball` | The one round trip that re-checks the path, runs `tar -xf`, and removes the scratch tarball. | the active startup step span | +| `sandbox.daytona.postUploadCommand` | One caller-supplied post-upload command. | the active startup step span | +| `sandbox.daytona.session.open` | The create of the one persistent session for a lease, on the first in-run command. | the active run span | +| `sandbox.daytona.session.close` | The delete of that persistent session on lease release. | the active run span | +| `sandbox.daytona.other` | Any span name outside the known set. | the active startup step span | + +The host clamps the span name to the closed set of leaf names above (`pack`, +`transfer`, `ensureDirectory`, `checkSymlinkEscape`, `promote`, `extractTarball`, +`postUploadCommand`, `session.open`, and `session.close`). The host maps a known +name to `sandbox.daytona.`. The host maps any other value to +`sandbox.daytona.other`, so a span name never carries free-form data. Only the +daytona provider emits these spans today, so the segment is the literal +`daytona`. + +The `sandbox.daytona.*` spans use this closed attribute allowlist. The host +drops every other key, so a command, an argument, a path, an id, a standard +output, or a standard error never rides a provider span. The host records only +the attributes that the producer sends for one span. + +| Attribute | Type | Optional | Meaning | +| --- | --- | --- | --- | +| `paperclip.sandbox.startup.provider` | string | no | The normalized provider family. | +| `paperclip.sandbox.startup.outcome` | string | yes | The step outcome (`ok`, `skipped`, or `failed`). | +| `paperclip.sandbox.startup.pack.wall_ms` | number | yes | The host-local wall time of the pack step. It rides the `sandbox.daytona.pack` span. | +| `paperclip.sandbox.startup.transfer.wall_ms` | number | yes | The wall time of the transfer step. It rides the `sandbox.daytona.transfer` span. | +| `paperclip.sandbox.startup.transfer.guard.count` | number | yes | The number of serial guard round trips before one transfer. It rides the `sandbox.daytona.transfer` span. | +| `paperclip.sandbox.startup.transfer.direction` | string | yes | The transfer direction (`inbound` or `outbound`). It rides the `sandbox.daytona.transfer` span. | + +The `span.record` host handler enforces the allowlist. It re-maps `provider` +through the provider-family normalizer. It keeps `outcome` only when the value +is `ok`, `skipped`, or `failed`. It keeps `transfer.direction` only when the +value is `inbound` or `outbound`. It keeps a numeric attribute only when the +value is a finite number. It drops a status message and keeps only the numeric +status code. The handler never throws, because observability must not change the +sync control flow. + +The `span.record` host method needs the `environment.drivers.register` +capability. So only a plugin that registers an environment driver may emit a +provider span. The capability gate rejects a provider span from any other +plugin. + +The host parents each provider span to the active sync task span. An inbound +transfer runs inside a `stage.*` task span, so its provider spans parent there. +An outbound transfer runs inside a `restore.*` task span under `sandbox.syncBack` +at teardown, so its provider spans parent there. The host mints a W3C +`traceparent` from the active task span and passes it to the plugin worker on the +per-call invocation channel. The teardown restore runs inside the run-parented +`sandbox.syncBack` span, so the host mints a `traceparent` for an outbound +provider span the same way it does for an inbound one. The worker tags its span with the +`traceparent` and treats the value as opaque. The worker never derives the +parent from it. The host recovers the `traceparent` from its own invocation +record, so a worker can never forge a parent. The host validates the +`traceparent` and rejects a missing or malformed value. With no active host +trace context the worker sends no span, so the whole provider-span path is a +no-op. + +## Sandbox Duplex Transport Instrumentation + +This section documents one duplex transport with three sinks: an +OpenTelemetry span, a counter in the `tool_runtime_metric_counters` table, and +one run-log event. + +Paperclip opens a fixed observability surface for the sandbox duplex transport. +This instrumentation is separate from Paperclip Telemetry events and from the +sandbox startup trace spans above. The generated telemetry contract does not +cover it, so this section is its canonical contract. The code owner is +`packages/adapter-utils/src/duplex-observability.ts`. That module holds each name and +each enum value as a literal constant, so the surface never drifts. + +The surface is opt-in. The host injects a recorder that binds the span to the +OTel tracer, the counter to the guarded counter store in +`server/src/services/tool-runtime-metrics.ts`, and the event to the run-events +bridge. The default recorder is a no-op, so the whole surface stays inert until +the host binds a real recorder. Every recorder call sits inside an error swallow, +so a telemetry failure never breaks the request path. + +The surface carries no user content. No route, no query, no request body, no +token, and no raw identifier rides a span, a counter, or an event. Each record +carries only the closed dimension keys below and, for the request span, a +latency. The `provider` dimension carries only the allowlisted public value +`daytona`. Any other plugin key maps to `other` before the record reaches a sink, +so a raw plugin key never reaches a span attribute, a counter label, or an event +field. + +### Spans + +| Span | Scope | Latency | +| --- | --- | --- | +| `sandbox.duplex.channel_open` | One duplex channel-open attempt. The `outcome` dimension is `ok` when the channel opened and readiness passed, or `error` when the open or readiness failed. | none | +| `sandbox.duplex.request` | One duplex request the broker forwarded to the host. | The request latency in milliseconds. | + +### Event + +| Event | Scope | +| --- | --- | +| `sandbox.duplex.transport` | The host emits it at each transport boundary: a ready duplex channel, a fallback to the file bridge, and a terminal channel loss. Its dimensions record the boundary. | + +### Counters + +| Counter | Scope | +| --- | --- | +| `sandbox_duplex_channel_open_total` | One successful duplex channel open. | +| `sandbox_duplex_fallback_total` | One fallback to the file bridge. The `fallback_reason` dimension records the cause. | +| `sandbox_duplex_loss_total` | One terminal duplex channel loss. The `loss_class` dimension records the phase. | +| `sandbox_duplex_session_leak_total` | One leaked provider session at teardown. | + +### Aggregate byte ledger metrics + +The host aggregate byte ledger owns one process-scoped gauge and two +process-scoped counters. The ledger bounds the retained bytes across every live +duplex route in one process. It sets the gauge on each reserve and each release. +It increments a counter on a rejected reservation and on an accounting defect. +These records carry no dimension label. The guarded counter store keys each +counter on `(companyId, metric)`, and the gauge reports one process value, so no +dynamic dimension rides them. The code owner is +`packages/adapter-utils/src/duplex-aggregate-byte-ledger.ts`, and the metric +names are literal constants in `duplex-observability.ts`. + +| Metric | Type | Scope | +| --- | --- | --- | +| `sandbox_duplex_aggregate_bytes_in_use` | gauge | The aggregate retained bytes across every live duplex route. The ledger sets it on each reserve and each release. | +| `sandbox_duplex_aggregate_byte_reservation_rejections_total` | counter | One rejected aggregate byte reservation. The ledger increments it when a reservation would pass the aggregate ceiling. | +| `sandbox_duplex_aggregate_byte_accounting_underflow_total` | counter | One aggregate byte accounting defect. The ledger increments it on a double release or on a transfer of a token it does not hold. | + +### Dimension keys + +Counters carry no dimension labels. The guarded counter store keys each counter +on `(companyId, metric)` with no label column, so the `fallback_reason` and +`loss_class` values fold into the counter metric name instead. The full closed +dimension set below rides only the spans and the `sandbox.duplex.transport` +event, which use only these closed keys. A test asserts the exact set, so a new +key never reaches a sink by accident. + +| Key | Type | Optional | Value set | +| --- | --- | --- | --- | +| `provider` | string | no | `daytona`, or `other` for any other plugin key. | +| `transport` | string | no | `duplex` or `file`. A fallback record uses `file`; every other record uses `duplex`. | +| `outcome` | string | yes | `ok` or `error`. | +| `fallback_reason` | string | yes | `gate_off`, `capability_absent`, `open_failed`, `ready_invalid`, `ready_nonce_mismatch`, `ready_timeout`, or `contaminated`. It rides only a fallback record. | +| `loss_class` | string | yes | `pre_dispatch` or `post_dispatch`, relative to the first request dispatch. It rides only a loss record. | +| `loss_reason` | string | yes | `stdin_eof`, `provider_exit`, `heartbeat_timeout`, `rpc_failure`, `write_error`, `transport_closed`, or `other`. The host maps every loss cause to one of these values, so no raw provider text reaches a sink. `write_error` marks a rejected host-to-sandbox write. `transport_closed` marks a reason-less provider transport close with no exit data. It rides only a loss record. | + +To add a name or an enum value, extend the literal constant in +`duplex-observability.ts` first, then update the test that asserts the closed set. +Keep every dimension low-cardinality and free of user content. diff --git a/doc/run-log-events.md b/doc/run-log-events.md new file mode 100644 index 0000000000..c7a55d80bd --- /dev/null +++ b/doc/run-log-events.md @@ -0,0 +1,78 @@ +# Run-Log Events + +Run-log events write to the `heartbeat_run_events` table +(`packages/db/src/schema/heartbeat_run_events.ts:6-20`). They are not +Paperclip Telemetry events, and they are not OpenTelemetry exports. A run-log +event needs no operator endpoint. + +## Sandbox Startup Run-Log Event + +Paperclip writes one `run.startup.step` event to the run log for each bring-up +step. This event is a run-log record, not a first-party telemetry event. The +generated telemetry contract does not cover it, so this section is its canonical +contract. + +The event payload carries only three fields. + +| Field | Type | Meaning | +| --- | --- | --- | +| `step` | string | The bring-up step name, for example `stage.sync`. | +| `durationMs` | number | The wall time of the step. A skipped step reports `0`. | +| `outcome` | string | The step outcome (`ok`, `skipped`, or `failed`). | + +The event no longer carries the per-step round-trip count or the provider +duration fields. It dropped `roundTrips`, `providerExecMs`, `providerGetMs`, +`createRuntimeMs`, and `ensureSessionMs`. The startup spans in +[`doc/observability.md`](observability.md) carry that detail now. The +`sandbox.exec` child spans hold the round-trip and provider durations. The +`acp.handshake` step span holds the create-runtime and ensure-session +sub-times. + +To read the detailed timing, use the startup spans. The spans need an OTLP +endpoint. A run with no endpoint keeps only the three run-log fields above. + +## Run Phase Timing Run-Log Event + +Paperclip writes one `run.phase.timing` event to the run log for each +run-lifecycle phase. This event is a run-log record, not a first-party telemetry +event. The generated telemetry contract does not cover it, so this section is its +canonical contract. The producer is `emitRunPhaseTiming` in +`packages/adapter-utils/src/acpx-engine/startup-timing.ts`. + +The event payload carries only three fields. + +| Field | Type | Meaning | +| --- | --- | --- | +| `phase` | string | The run-lifecycle phase name from the closed allowlist below. | +| `durationMs` | number | The wall time of the phase. A negative or a non-finite value clamps to `0`. | +| `outcome` | string | The phase outcome (`ok` or `failed`). | + +The `phase` field is one member of a closed, low-cardinality allowlist. The +producer drops any event whose phase name is outside this allowlist, so a +free-form label never reaches the run log. The allowlist has twelve phase names. + +| Phase | Meaning | +| --- | --- | +| `place_workspace` | Place the run workspace. | +| `start_transport` | Start the agent transport. | +| `create_runtime` | Create the agent runtime. | +| `ensure_session` | Ensure the agent session exists. | +| `configure_session` | Configure the agent session. | +| `prepare_turn` | Prepare the turn. | +| `turn` | Run the turn. | +| `end_session` | End the agent session. | +| `settle_reuse` | Settle the session for reuse. | +| `stop_transport` | Stop the agent transport. | +| `sync_back` | Sync the workspace back. | +| `release_staging_lease` | Release the staging lease. | + +The payload never carries a command, an argument, a path, an environment value, +or a raw identifier. The event rides the `ctx.onEvent` run-event bridge and is +run-log-only. It needs no OTLP endpoint. + +## Related instrumentation + +The sandbox duplex transport also writes one run-log event as one of its three +sinks. See the +[Sandbox Duplex Transport Instrumentation](observability.md#sandbox-duplex-transport-instrumentation) +section in the Observability contract. diff --git a/packages/adapter-utils/src/acpx-engine/execute.ts b/packages/adapter-utils/src/acpx-engine/execute.ts index 45a73f34b9..925f01ae36 100644 --- a/packages/adapter-utils/src/acpx-engine/execute.ts +++ b/packages/adapter-utils/src/acpx-engine/execute.ts @@ -15,7 +15,7 @@ import type { import { adapterExecutionTargetSessionIdentity, describeAdapterExecutionTarget, - adapterExecutionTargetDuplexTelemetryRecorder, + adapterExecutionTargetDuplexObservabilityRecorder, adapterExecutionTargetEnablesSandboxDuplexBridge, formatAdapterExecutionTimeoutErrorMessage, formatAdapterExecutionTimeoutStartLogLine, @@ -33,7 +33,7 @@ import { type PreparedAdapterExecutionTargetRuntime, type SandboxAdditionalSource, } from "@paperclipai/adapter-utils/execution-target"; -import type { DuplexLossReason } from "../duplex-telemetry.js"; +import type { DuplexLossReason } from "../duplex-observability.js"; import { DUPLEX_CHANNEL_LOST_ERROR_CODE } from "../duplex-bridge-broker.js"; import { DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE, @@ -2052,7 +2052,7 @@ async function buildRuntime(input: { timeoutSec, hostApiToken: env.PAPERCLIP_API_KEY, enableSandboxDuplexBridge: adapterExecutionTargetEnablesSandboxDuplexBridge(remoteTarget), - duplexTelemetryRecorder: adapterExecutionTargetDuplexTelemetryRecorder(remoteTarget), + duplexObservabilityRecorder: adapterExecutionTargetDuplexObservabilityRecorder(remoteTarget), onLog: input.ctx.onLog, getRuntimeParentContext: input.getRuntimeParentContext, runtimeSpan: input.runtimeSpan, @@ -3390,10 +3390,11 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) { .onLog("stderr", `[paperclip] ACPX teardown step "${step}" failed: ${reason}\n`) .catch(() => {}); }; - // Emit one per-phase timing telemetry event. It is observability-only: it - // carries the phase name (from the closed allowlist), the wall time, and the - // outcome, and never a command, a path, an environment value, or an - // identifier. Telemetry failure never fails the run. + // Emit one per-phase timing run-log event. It is not an OpenTelemetry + // export and it is not a Telemetry event: it carries the phase name + // (from the closed allowlist), the wall time, and the outcome, and + // never a command, a path, an environment value, or an identifier. A + // failure to emit this event never fails the run. const emitPhase = (phase: string, startMs: number, outcome: "ok" | "failed"): Promise => emitRunPhaseTiming(ctx, phase, now() - startMs, outcome); // Time a settlement step and emit its phase timing on every path. A step diff --git a/packages/adapter-utils/src/acpx-engine/startup-timing.ts b/packages/adapter-utils/src/acpx-engine/startup-timing.ts index 131c7cf621..ebbd237bef 100644 --- a/packages/adapter-utils/src/acpx-engine/startup-timing.ts +++ b/packages/adapter-utils/src/acpx-engine/startup-timing.ts @@ -43,7 +43,7 @@ export function normalizeProviderFamily(key: string | undefined): string { /** * The common prefix for every sandbox-startup span attribute. One prefix keeps - * the attribute namespace closed and easy to find in the telemetry backend. + * the attribute namespace closed and easy to find in the OpenTelemetry backend. */ export const SANDBOX_STARTUP_SPAN_ATTR_PREFIX = "paperclip.sandbox.startup."; @@ -794,11 +794,12 @@ export async function emitSkippedStartupStep( /** * Structured event emitted once per named run-lifecycle phase, so the duration - * and the outcome of each phase land in the run-events stream. It is - * observability-only and rides the existing `ctx.onEvent` bridge. The payload is - * a closed shape: exactly `phase`, `durationMs`, and `outcome`. The phase name is - * from a closed allowlist, so the event never carries a command, an argument, a - * path, an environment value, or a raw identifier. + * and the outcome of each phase land in the run-events stream. It is a + * run-log event and rides the existing `ctx.onEvent` bridge. It never changes + * startup control flow. The payload is a closed shape: exactly `phase`, + * `durationMs`, and `outcome`. The phase name is from a closed allowlist, so + * the event never carries a command, an argument, a path, an environment + * value, or a raw identifier. */ export const RUN_PHASE_TIMING_EVENT_TYPE = "run.phase.timing"; diff --git a/packages/adapter-utils/src/duplex-aggregate-byte-ledger.ts b/packages/adapter-utils/src/duplex-aggregate-byte-ledger.ts index e7f7aec3c1..b544dee9f1 100644 --- a/packages/adapter-utils/src/duplex-aggregate-byte-ledger.ts +++ b/packages/adapter-utils/src/duplex-aggregate-byte-ledger.ts @@ -34,7 +34,7 @@ import { DUPLEX_COUNTER_AGGREGATE_BYTE_ACCOUNTING_UNDERFLOW_TOTAL, DUPLEX_COUNTER_AGGREGATE_BYTE_RESERVATION_REJECTIONS_TOTAL, DUPLEX_GAUGE_AGGREGATE_BYTES_IN_USE, -} from "./duplex-telemetry.js"; +} from "./duplex-observability.js"; /** * The public rejection marker. A retention site that cannot reserve its bytes diff --git a/packages/adapter-utils/src/duplex-bridge-broker.test.ts b/packages/adapter-utils/src/duplex-bridge-broker.test.ts index 346c0cbf39..57cb89d1bb 100644 --- a/packages/adapter-utils/src/duplex-bridge-broker.test.ts +++ b/packages/adapter-utils/src/duplex-bridge-broker.test.ts @@ -16,13 +16,13 @@ import { } from "./duplex-frame-codec.js"; import { splitBodyIntoChunkFrames } from "./duplex-body-spool.js"; import { - createDuplexTelemetry, + createDuplexObservability, DUPLEX_COUNTER_LOSS_TOTAL, DUPLEX_SPAN_REQUEST, - type DuplexTelemetryCounterRecord, - type DuplexTelemetryEventRecord, - type DuplexTelemetrySpanRecord, -} from "./duplex-telemetry.js"; + type DuplexObservabilityCounterRecord, + type DuplexObservabilityEventRecord, + type DuplexObservabilitySpanRecord, +} from "./duplex-observability.js"; import type { CommandManagedDuplexChannel } from "./command-managed-runtime.js"; /** @@ -199,15 +199,15 @@ async function flush(): Promise { /** The telemetry sink capture. It proves the broker records nothing for a refusal. */ interface TelemetryCapture { - spans: DuplexTelemetrySpanRecord[]; - counters: DuplexTelemetryCounterRecord[]; - events: DuplexTelemetryEventRecord[]; + spans: DuplexObservabilitySpanRecord[]; + counters: DuplexObservabilityCounterRecord[]; + events: DuplexObservabilityEventRecord[]; } /** Build the real telemetry facade over a capturing recorder. */ -function createTelemetryCapture(): { telemetry: ReturnType; capture: TelemetryCapture } { +function createTelemetryCapture(): { telemetry: ReturnType; capture: TelemetryCapture } { const capture: TelemetryCapture = { spans: [], counters: [], events: [] }; - const telemetry = createDuplexTelemetry({ + const telemetry = createDuplexObservability({ providerKey: "daytona", recorder: { recordSpan: (record) => capture.spans.push(record), diff --git a/packages/adapter-utils/src/duplex-bridge-broker.ts b/packages/adapter-utils/src/duplex-bridge-broker.ts index 1ae50d92bb..00ebf516bb 100644 --- a/packages/adapter-utils/src/duplex-bridge-broker.ts +++ b/packages/adapter-utils/src/duplex-bridge-broker.ts @@ -71,8 +71,8 @@ import { import type { DuplexLossReason, DuplexOutcomeValue, - DuplexTelemetry, -} from "./duplex-telemetry.js"; + DuplexObservability, +} from "./duplex-observability.js"; /** The lifecycle states of the broker. The broker moves through them in order. */ export type DuplexBrokerState = "opening" | "open" | "lost" | "closing" | "closed"; @@ -285,7 +285,7 @@ export interface DuplexBrokerOptions { * record to the fixed names and dimensions, so no route, query, body, token, or * raw error rides a span or a counter. The default records nothing. */ - telemetry?: DuplexTelemetry; + telemetry?: DuplexObservability; /** * The process-owned aggregate byte ledger. The broker reserves the exact retained * bytes of each dispatched request against it before it retains the frame: the diff --git a/packages/adapter-utils/src/duplex-telemetry.ts b/packages/adapter-utils/src/duplex-observability.ts similarity index 89% rename from packages/adapter-utils/src/duplex-telemetry.ts rename to packages/adapter-utils/src/duplex-observability.ts index b31c655918..d88769a4a3 100644 --- a/packages/adapter-utils/src/duplex-telemetry.ts +++ b/packages/adapter-utils/src/duplex-observability.ts @@ -12,7 +12,7 @@ * never reaches a span attribute, a counter label, or an event field. * * The module stays free of `@opentelemetry/api` and of the database. The host - * injects a {@link DuplexTelemetryRecorder}; the default is a no-op recorder, so + * injects a {@link DuplexObservabilityRecorder}; the default is a no-op recorder, so * the whole surface stays inert until the host binds a real recorder. Every * recorder call sits inside an error swallow, so a telemetry failure never breaks * the request path. @@ -59,9 +59,9 @@ export const DUPLEX_COUNTER_AGGREGATE_BYTE_ACCOUNTING_UNDERFLOW_TOTAL = /** * The closed set of aggregate byte ledger metric names. A test pins this exact * set, so a new ledger metric name needs an explicit review. Each record uses - * only closed constant dimensions and no dynamic label. The telemetry contract - * documents these metrics under "Aggregate byte ledger metrics" in - * `packages/shared/src/telemetry/README.md`. + * only closed constant dimensions and no dynamic label. The Observability + * contract documents these metrics under "Aggregate byte ledger metrics" in + * `doc/observability.md`. */ export const DUPLEX_AGGREGATE_BYTE_LEDGER_METRIC_NAMES = [ DUPLEX_GAUGE_AGGREGATE_BYTES_IN_USE, @@ -182,7 +182,7 @@ export function normalizeDuplexProvider(key: string | null | undefined): DuplexP * are present only when the record defines them, so a span or a counter never * carries an empty dimension. */ -export interface DuplexTelemetryDimensions { +export interface DuplexObservabilityDimensions { provider: DuplexProviderValue; transport: DuplexTransportValue; outcome?: DuplexOutcomeValue; @@ -192,23 +192,23 @@ export interface DuplexTelemetryDimensions { } /** One span record the host records. The request span carries a latency. */ -export interface DuplexTelemetrySpanRecord { +export interface DuplexObservabilitySpanRecord { name: string; - dimensions: DuplexTelemetryDimensions; + dimensions: DuplexObservabilityDimensions; /** The request latency in milliseconds. Only the request span sets it. */ latencyMs?: number; } /** One counter increment the host records. */ -export interface DuplexTelemetryCounterRecord { +export interface DuplexObservabilityCounterRecord { metric: string; - dimensions: DuplexTelemetryDimensions; + dimensions: DuplexObservabilityDimensions; } /** One event the host emits. */ -export interface DuplexTelemetryEventRecord { +export interface DuplexObservabilityEventRecord { name: string; - dimensions: DuplexTelemetryDimensions; + dimensions: DuplexObservabilityDimensions; } /** @@ -218,14 +218,14 @@ export interface DuplexTelemetryEventRecord { * The recorder receives only already-mapped dimensions, so the raw provider key * never reaches it. */ -export interface DuplexTelemetryRecorder { - recordSpan(record: DuplexTelemetrySpanRecord): void; - incrementCounter(record: DuplexTelemetryCounterRecord): void; - emitEvent(record: DuplexTelemetryEventRecord): void; +export interface DuplexObservabilityRecorder { + recordSpan(record: DuplexObservabilitySpanRecord): void; + incrementCounter(record: DuplexObservabilityCounterRecord): void; + emitEvent(record: DuplexObservabilityEventRecord): void; } /** A no-op recorder. Every method does nothing, so the surface stays inert. */ -export const NOOP_DUPLEX_TELEMETRY_RECORDER: DuplexTelemetryRecorder = { +export const NOOP_DUPLEX_OBSERVABILITY_RECORDER: DuplexObservabilityRecorder = { recordSpan() {}, incrementCounter() {}, emitEvent() {}, @@ -252,7 +252,7 @@ export interface DuplexChannelOpenAttempt { * provider, so a call site never passes a raw key. It maps each semantic event to * the fixed names and dimensions, then calls the recorder inside an error swallow. */ -export interface DuplexTelemetry { +export interface DuplexObservability { /** Begin a channel-open attempt. The caller reports `ready` or `fallback`. */ startChannelOpen(): DuplexChannelOpenAttempt; /** @@ -272,10 +272,10 @@ export interface DuplexTelemetry { recordSessionLeak(): void; } -/** The options for {@link createDuplexTelemetry}. */ -export interface DuplexTelemetryOptions { +/** The options for {@link createDuplexObservability}. */ +export interface DuplexObservabilityOptions { /** The injected recorder. The default is the no-op recorder. */ - recorder?: DuplexTelemetryRecorder | null; + recorder?: DuplexObservabilityRecorder | null; /** The raw provider key. The facade maps it through the allowlist one time. */ providerKey?: string | null; } @@ -286,25 +286,25 @@ export interface DuplexTelemetryOptions { * inside a `try/catch`, so a throwing recorder never breaks the request path. A * missing recorder yields a facade whose methods do nothing. */ -export function createDuplexTelemetry(options: DuplexTelemetryOptions = {}): DuplexTelemetry { - const recorder = options.recorder ?? NOOP_DUPLEX_TELEMETRY_RECORDER; +export function createDuplexObservability(options: DuplexObservabilityOptions = {}): DuplexObservability { + const recorder = options.recorder ?? NOOP_DUPLEX_OBSERVABILITY_RECORDER; const provider = normalizeDuplexProvider(options.providerKey); - const safeSpan = (record: DuplexTelemetrySpanRecord): void => { + const safeSpan = (record: DuplexObservabilitySpanRecord): void => { try { recorder.recordSpan(record); } catch { // Observability must not break the request path. } }; - const safeCounter = (record: DuplexTelemetryCounterRecord): void => { + const safeCounter = (record: DuplexObservabilityCounterRecord): void => { try { recorder.incrementCounter(record); } catch { // Observability must not break the request path. } }; - const safeEvent = (record: DuplexTelemetryEventRecord): void => { + const safeEvent = (record: DuplexObservabilityEventRecord): void => { try { recorder.emitEvent(record); } catch { @@ -313,7 +313,7 @@ export function createDuplexTelemetry(options: DuplexTelemetryOptions = {}): Dup }; const recordFallback = (reason: DuplexFallbackReason): void => { - const dimensions: DuplexTelemetryDimensions = { + const dimensions: DuplexObservabilityDimensions = { provider, transport: "file", outcome: "error", @@ -330,7 +330,7 @@ export function createDuplexTelemetry(options: DuplexTelemetryOptions = {}): Dup ready(): void { if (settled) return; settled = true; - const dimensions: DuplexTelemetryDimensions = { + const dimensions: DuplexObservabilityDimensions = { provider, transport: "duplex", outcome: "ok", @@ -364,7 +364,7 @@ export function createDuplexTelemetry(options: DuplexTelemetryOptions = {}): Dup }); }, recordLoss(lossClass: DuplexLossClass, lossReason: DuplexLossReason): void { - const dimensions: DuplexTelemetryDimensions = { + const dimensions: DuplexObservabilityDimensions = { provider, transport: "duplex", outcome: "error", diff --git a/packages/adapter-utils/src/execution-target-sandbox.test.ts b/packages/adapter-utils/src/execution-target-sandbox.test.ts index b5c5c6b148..5c3f86c03a 100644 --- a/packages/adapter-utils/src/execution-target-sandbox.test.ts +++ b/packages/adapter-utils/src/execution-target-sandbox.test.ts @@ -17,7 +17,7 @@ import { __duplexReadinessTesting, buildDuplexGatewayLaunchArgv, DEFAULT_REMOTE_SANDBOX_ADAPTER_TIMEOUT_SEC, - adapterExecutionTargetDuplexTelemetryRecorder, + adapterExecutionTargetDuplexObservabilityRecorder, adapterExecutionTargetEnablesSandboxDuplexBridge, adapterExecutionTargetSessionIdentity, adapterExecutionTargetToRemoteSpec, @@ -72,7 +72,7 @@ import { type DuplexBrokerState, } from "./duplex-bridge-broker.js"; import { - createDuplexTelemetry, + createDuplexObservability, DUPLEX_AGGREGATE_BYTE_LEDGER_METRIC_NAMES, DUPLEX_COUNTER_AGGREGATE_BYTE_ACCOUNTING_UNDERFLOW_TOTAL, DUPLEX_COUNTER_AGGREGATE_BYTE_RESERVATION_REJECTIONS_TOTAL, @@ -84,12 +84,12 @@ import { DUPLEX_SPAN_CHANNEL_OPEN, DUPLEX_SPAN_REQUEST, DUPLEX_TRANSPORT_EVENT, - type DuplexTelemetryCounterRecord, - type DuplexTelemetryDimensions, - type DuplexTelemetryEventRecord, - type DuplexTelemetryRecorder, - type DuplexTelemetrySpanRecord, -} from "./duplex-telemetry.js"; + type DuplexObservabilityCounterRecord, + type DuplexObservabilityDimensions, + type DuplexObservabilityEventRecord, + type DuplexObservabilityRecorder, + type DuplexObservabilitySpanRecord, +} from "./duplex-observability.js"; const execFileAsync = promisify(execFile); @@ -3374,7 +3374,7 @@ describe("sandbox adapter execution targets", () => { hostApiToken: "real-run-jwt", hostApiUrl: api.origin, enableSandboxDuplexBridge: true, - duplexTelemetryRecorder: recorder, + duplexObservabilityRecorder: recorder, }); try { // The file bridge serves, not the duplex transport. @@ -3485,8 +3485,8 @@ describe("sandbox adapter execution targets", () => { await mkdir(remoteCwd, { recursive: true }); const api = await startRecordingApiServer(); - const counters: DuplexTelemetryCounterRecord[] = []; - const recorder: DuplexTelemetryRecorder = { + const counters: DuplexObservabilityCounterRecord[] = []; + const recorder: DuplexObservabilityRecorder = { recordSpan() {}, incrementCounter(record) { counters.push(record); @@ -3505,7 +3505,7 @@ describe("sandbox adapter execution targets", () => { timeoutMs: 30_000, runner: makeDuplexSelectionRunner().runner, effectiveCapabilities: duplexCapabilities(true), - duplexTelemetryRecorder: recorder, + duplexObservabilityRecorder: recorder, }; const openBridge = await startAdapterExecutionTargetPaperclipBridge({ runId: "run-duplex-open", @@ -3515,7 +3515,7 @@ describe("sandbox adapter execution targets", () => { hostApiToken: "real-run-jwt", hostApiUrl: api.origin, enableSandboxDuplexBridge: true, - duplexTelemetryRecorder: adapterExecutionTargetDuplexTelemetryRecorder(openTarget), + duplexObservabilityRecorder: adapterExecutionTargetDuplexObservabilityRecorder(openTarget), }); try { expect(openBridge?.env.PAPERCLIP_API_BRIDGE_MODE).toBe("duplex_v1"); @@ -3537,7 +3537,7 @@ describe("sandbox adapter execution targets", () => { timeoutMs: 30_000, runner: makeDuplexSelectionRunner().runner, effectiveCapabilities: duplexCapabilities(true), - duplexTelemetryRecorder: recorder, + duplexObservabilityRecorder: recorder, }; const fallbackBridge = await startAdapterExecutionTargetPaperclipBridge({ runId: "run-duplex-fallback", @@ -3547,7 +3547,7 @@ describe("sandbox adapter execution targets", () => { hostApiToken: "real-run-jwt", hostApiUrl: api.origin, enableSandboxDuplexBridge: false, - duplexTelemetryRecorder: adapterExecutionTargetDuplexTelemetryRecorder(fallbackTarget), + duplexObservabilityRecorder: adapterExecutionTargetDuplexObservabilityRecorder(fallbackTarget), }); try { expect(fallbackBridge?.env.PAPERCLIP_API_BRIDGE_MODE).toBe("queue_v1"); @@ -3803,15 +3803,15 @@ describe("sandbox adapter execution targets", () => { // dimensions, and values. An optional `failEvery` flag makes every method throw, // so a test proves a telemetry failure never breaks the request path. function createRecordingDuplexRecorder(options: { failEvery?: boolean } = {}): { - recorder: DuplexTelemetryRecorder; - spans: DuplexTelemetrySpanRecord[]; - counters: DuplexTelemetryCounterRecord[]; - events: DuplexTelemetryEventRecord[]; + recorder: DuplexObservabilityRecorder; + spans: DuplexObservabilitySpanRecord[]; + counters: DuplexObservabilityCounterRecord[]; + events: DuplexObservabilityEventRecord[]; } { - const spans: DuplexTelemetrySpanRecord[] = []; - const counters: DuplexTelemetryCounterRecord[] = []; - const events: DuplexTelemetryEventRecord[] = []; - const recorder: DuplexTelemetryRecorder = { + const spans: DuplexObservabilitySpanRecord[] = []; + const counters: DuplexObservabilityCounterRecord[] = []; + const events: DuplexObservabilityEventRecord[] = []; + const recorder: DuplexObservabilityRecorder = { recordSpan(record) { if (options.failEvery) throw new Error("telemetry sink down"); spans.push(record); @@ -3830,7 +3830,7 @@ describe("sandbox adapter execution targets", () => { // Every dimension key a record carries must be one of the fixed keys. The set is // closed, so a new key never reaches a sink by accident. - function assertOnlyFixedDimensionKeys(dimensions: DuplexTelemetryDimensions | undefined): void { + function assertOnlyFixedDimensionKeys(dimensions: DuplexObservabilityDimensions | undefined): void { expect(dimensions).toBeDefined(); for (const key of Object.keys(dimensions ?? {})) { expect(DUPLEX_DIMENSION_KEYS).toContain(key as (typeof DUPLEX_DIMENSION_KEYS)[number]); @@ -3894,7 +3894,7 @@ describe("sandbox adapter execution targets", () => { hostApiToken: "real-run-jwt", hostApiUrl: api.origin, enableSandboxDuplexBridge: true, - duplexTelemetryRecorder: recorder, + duplexObservabilityRecorder: recorder, }); try { expect(bridge?.env.PAPERCLIP_API_BRIDGE_MODE).toBe("duplex_v1"); @@ -3970,7 +3970,7 @@ describe("sandbox adapter execution targets", () => { hostApiToken: "real-run-jwt", hostApiUrl: api.origin, enableSandboxDuplexBridge: true, - duplexTelemetryRecorder: recorder, + duplexObservabilityRecorder: recorder, }); try { expect(bridge?.env.PAPERCLIP_API_BRIDGE_MODE).toBe("queue_v1"); @@ -4049,7 +4049,7 @@ describe("sandbox adapter execution targets", () => { hostApiToken: "real-run-jwt", hostApiUrl: api.origin, enableSandboxDuplexBridge: true, - duplexTelemetryRecorder: recorder, + duplexObservabilityRecorder: recorder, }); try { // The channel never opened, so the host serves the file bridge. @@ -4104,7 +4104,7 @@ describe("sandbox adapter execution targets", () => { hostApiToken: "real-run-jwt", hostApiUrl: api.origin, enableSandboxDuplexBridge: true, - duplexTelemetryRecorder: recorder, + duplexObservabilityRecorder: recorder, }); try { expect(bridge?.env.PAPERCLIP_API_BRIDGE_MODE).toBe("duplex_v1"); @@ -4171,7 +4171,7 @@ describe("sandbox adapter execution targets", () => { hostApiToken: "real-run-jwt", hostApiUrl: api.origin, enableSandboxDuplexBridge: true, - duplexTelemetryRecorder: recorder, + duplexObservabilityRecorder: recorder, }); try { // The throwing recorder never blocked the duplex selection. @@ -4279,7 +4279,7 @@ describe("sandbox adapter execution targets", () => { hostApiToken: AGENT_TOKEN_SENTINEL, hostApiUrl: api.origin, enableSandboxDuplexBridge: true, - duplexTelemetryRecorder: recorder, + duplexObservabilityRecorder: recorder, onLog: async (_stream, chunk) => { logLines.push(chunk); }, @@ -4351,7 +4351,7 @@ describe("sandbox adapter execution targets", () => { hostApiToken: "real-run-jwt", hostApiUrl: api.origin, enableSandboxDuplexBridge: true, - duplexTelemetryRecorder: recorder, + duplexObservabilityRecorder: recorder, }); try { expect(bridge?.env.PAPERCLIP_API_BRIDGE_MODE).toBe("duplex_v1"); @@ -4430,7 +4430,7 @@ describe("sandbox adapter execution targets", () => { // A long readiness timeout, so the buffer cap, not the timeout, drives the // failure. duplexReadinessTimeoutMs: 5_000, - duplexTelemetryRecorder: recorder, + duplexObservabilityRecorder: recorder, }); try { expect(bridge).not.toBeNull(); @@ -4488,7 +4488,7 @@ describe("sandbox adapter execution targets", () => { // A long readiness timeout, so the buffer cap, not the timeout, drives the // failure. duplexReadinessTimeoutMs: 5_000, - duplexTelemetryRecorder: recorder, + duplexObservabilityRecorder: recorder, }); try { expect(bridge).not.toBeNull(); @@ -4544,7 +4544,7 @@ describe("sandbox adapter execution targets", () => { // A long readiness timeout, so the aggregate ledger, not the timeout, drives // the failure. duplexReadinessTimeoutMs: 5_000, - duplexTelemetryRecorder: recorder, + duplexObservabilityRecorder: recorder, }); try { expect(bridge).not.toBeNull(); @@ -4666,7 +4666,7 @@ describe("sandbox adapter execution targets", () => { hostApiUrl: api.origin, enableSandboxDuplexBridge: true, duplexReadinessTimeoutMs: 5_000, - duplexTelemetryRecorder: recorder, + duplexObservabilityRecorder: recorder, }); try { expect(bridge).not.toBeNull(); @@ -4722,7 +4722,7 @@ describe("sandbox adapter execution targets", () => { hostApiUrl: api.origin, enableSandboxDuplexBridge: true, duplexReadinessTimeoutMs: 5_000, - duplexTelemetryRecorder: recorder, + duplexObservabilityRecorder: recorder, }); try { expect(bridge).not.toBeNull(); @@ -4777,7 +4777,7 @@ describe("sandbox adapter execution targets", () => { hostApiUrl: api.origin, enableSandboxDuplexBridge: true, duplexReadinessTimeoutMs: 5_000, - duplexTelemetryRecorder: recorder, + duplexObservabilityRecorder: recorder, }); try { expect(bridge).not.toBeNull(); @@ -4827,7 +4827,7 @@ describe("sandbox adapter execution targets", () => { hostApiUrl: api.origin, enableSandboxDuplexBridge: true, duplexReadinessTimeoutMs: 5_000, - duplexTelemetryRecorder: recorder, + duplexObservabilityRecorder: recorder, }); try { expect(bridge).not.toBeNull(); @@ -4885,7 +4885,7 @@ describe("sandbox adapter execution targets", () => { enableSandboxDuplexBridge: true, // A long readiness timeout, so the cap, not the timeout, drives the failure. duplexReadinessTimeoutMs: 5_000, - duplexTelemetryRecorder: recorder, + duplexObservabilityRecorder: recorder, }); try { expect(bridge).not.toBeNull(); @@ -4935,7 +4935,7 @@ describe("sandbox adapter execution targets", () => { hostApiUrl: api.origin, enableSandboxDuplexBridge: true, duplexReadinessTimeoutMs: 5_000, - duplexTelemetryRecorder: recorder, + duplexObservabilityRecorder: recorder, }); try { expect(bridge?.env.PAPERCLIP_API_BRIDGE_MODE).toBe("queue_v1"); @@ -6311,9 +6311,9 @@ describe("createDuplexBridgeBroker", () => { }); it("keeps a success when a loss orders after a host-observed orderly completion, and emits no loss event", async () => { - const events: DuplexTelemetryEventRecord[] = []; - const counters: DuplexTelemetryCounterRecord[] = []; - const recorder: DuplexTelemetryRecorder = { + const events: DuplexObservabilityEventRecord[] = []; + const counters: DuplexObservabilityCounterRecord[] = []; + const recorder: DuplexObservabilityRecorder = { recordSpan() {}, incrementCounter(record) { counters.push(record); @@ -6326,7 +6326,7 @@ describe("createDuplexBridgeBroker", () => { const broker = await createDuplexBridgeBroker({ channel: fake.channel, forwardRequest: async () => ({ status: 200 }), - telemetry: createDuplexTelemetry({ recorder, providerKey: "daytona" }), + telemetry: createDuplexObservability({ recorder, providerKey: "daytona" }), }); broker.start(); @@ -6342,10 +6342,10 @@ describe("createDuplexBridgeBroker", () => { }); it("carries the typed loss_reason on the transport loss event and keeps a sentinel message off every sink", async () => { - const spans: DuplexTelemetrySpanRecord[] = []; - const counters: DuplexTelemetryCounterRecord[] = []; - const events: DuplexTelemetryEventRecord[] = []; - const recorder: DuplexTelemetryRecorder = { + const spans: DuplexObservabilitySpanRecord[] = []; + const counters: DuplexObservabilityCounterRecord[] = []; + const events: DuplexObservabilityEventRecord[] = []; + const recorder: DuplexObservabilityRecorder = { recordSpan(record) { spans.push(record); }, @@ -6362,7 +6362,7 @@ describe("createDuplexBridgeBroker", () => { const broker = await createDuplexBridgeBroker({ channel: fake.channel, forwardRequest: async () => ({ status: 200 }), - telemetry: createDuplexTelemetry({ recorder, providerKey: "daytona" }), + telemetry: createDuplexObservability({ recorder, providerKey: "daytona" }), logger: (message) => logLines.push(message), }); broker.start(); diff --git a/packages/adapter-utils/src/execution-target.ts b/packages/adapter-utils/src/execution-target.ts index c21b7d3a87..a974f39453 100644 --- a/packages/adapter-utils/src/execution-target.ts +++ b/packages/adapter-utils/src/execution-target.ts @@ -62,10 +62,10 @@ import { } from "./duplex-frame-codec.js"; import type { ReassembledBody } from "./duplex-body-spool.js"; import { - createDuplexTelemetry, + createDuplexObservability, type DuplexFallbackReason, - type DuplexTelemetryRecorder, -} from "./duplex-telemetry.js"; + type DuplexObservabilityRecorder, +} from "./duplex-observability.js"; import { DUPLEX_CHANNEL_AGGREGATE_BYTES_EXCEEDED, type DuplexAggregateByteLedger, @@ -186,12 +186,12 @@ export interface AdapterSandboxExecutionTarget extends AdapterExecutionTargetWor */ streamRunLogs?: boolean | null; /** - * The injected duplex telemetry recorder for this run. The host attaches it on - * the same seam as `runner`, so this live object stays on the host and never - * enters the sandbox environment. The bridge binds it to the fixed duplex - * observability surface. Absent means the safe no-op default. + * The injected duplex observability recorder for this run. The host attaches + * it on the same seam as `runner`, so this live object stays on the host and + * never enters the sandbox environment. The bridge binds it to the fixed + * duplex observability surface. Absent means the safe no-op default. */ - duplexTelemetryRecorder?: DuplexTelemetryRecorder | null; + duplexObservabilityRecorder?: DuplexObservabilityRecorder | null; /** * The process-owned aggregate byte ledger for the sandbox duplex channel. The * host stamps this same object on every sandbox target on the same seam as @@ -434,15 +434,15 @@ export function adapterExecutionTargetEnablesSandboxDuplexBridge( } /** - * Read the injected duplex telemetry recorder off a target. Only a sandbox - * target with a recorder attached returns it. Every other target returns null, - * so the bridge falls back to the safe no-op recorder. + * Read the injected duplex observability recorder off a target. Only a + * sandbox target with a recorder attached returns it. Every other target + * returns null, so the bridge falls back to the safe no-op recorder. */ -export function adapterExecutionTargetDuplexTelemetryRecorder( +export function adapterExecutionTargetDuplexObservabilityRecorder( target: AdapterExecutionTarget | null | undefined, -): DuplexTelemetryRecorder | null { +): DuplexObservabilityRecorder | null { return target?.kind === "remote" && target.transport === "sandbox" - ? target.duplexTelemetryRecorder ?? null + ? target.duplexObservabilityRecorder ?? null : null; } @@ -3045,7 +3045,7 @@ export async function startAdapterExecutionTargetPaperclipBridge(input: { // span, the request span, the guarded counters, and the transport event. The // default is a no-op recorder, so the surface stays inert until the host injects // a real recorder. - duplexTelemetryRecorder?: DuplexTelemetryRecorder | null; + duplexObservabilityRecorder?: DuplexObservabilityRecorder | null; }): Promise { if (!adapterExecutionTargetUsesPaperclipBridge(input.target)) { return null; @@ -3116,8 +3116,8 @@ export async function startAdapterExecutionTargetPaperclipBridge(input: { // is a no-op, so the facade is inert until the host injects a real recorder. const duplexProviderKey = "providerKey" in target ? target.providerKey ?? undefined : undefined; - const duplexTelemetry = createDuplexTelemetry({ - recorder: input.duplexTelemetryRecorder ?? undefined, + const duplexObservability = createDuplexObservability({ + recorder: input.duplexObservabilityRecorder ?? undefined, providerKey: duplexProviderKey, }); @@ -3270,14 +3270,14 @@ export async function startAdapterExecutionTargetPaperclipBridge(input: { // capability or the runner method absent. A later channel-open failure records // its own fallback through the channel-open attempt below. if (!duplexRequested) { - duplexTelemetry.recordFallback("gate_off"); + duplexObservability.recordFallback("gate_off"); } else if (!capabilityGranted || typeof openDuplexChannel !== "function") { - duplexTelemetry.recordFallback("capability_absent"); + duplexObservability.recordFallback("capability_absent"); } if (duplexRequested && capabilityGranted && typeof openDuplexChannel === "function") { // Begin the channel-open attempt. The block reports exactly one terminal: // `ready` on success, or `fallback(reason)` on an open or a readiness failure. - const duplexChannelOpen = duplexTelemetry.startChannelOpen(); + const duplexChannelOpen = duplexObservability.startChannelOpen(); const readinessTimeoutMs = typeof input.duplexReadinessTimeoutMs === "number" && Number.isFinite(input.duplexReadinessTimeoutMs) && @@ -3442,7 +3442,7 @@ export async function startAdapterExecutionTargetPaperclipBridge(input: { }, // The duplex path emits only the fixed transport telemetry. It passes no // free-form logger, so no raw provider error rides a log line here. - telemetry: duplexTelemetry, + telemetry: duplexObservability, // Surface a terminal channel loss on the run log. The broker latches // the failure on its ordered lifecycle; the host names only the typed, // closed loss reason here, never the raw provider message. The caller @@ -3532,7 +3532,7 @@ export async function startAdapterExecutionTargetPaperclipBridge(input: { // A channel that did not reach the `closed` state may leave a live // provider session, so record one session leak. if (activeBroker.state !== "closed") { - duplexTelemetry.recordSessionLeak(); + duplexObservability.recordSessionLeak(); } await bridgeAsset.cleanup(); }, diff --git a/packages/adapter-utils/src/published-exports.test.ts b/packages/adapter-utils/src/published-exports.test.ts new file mode 100644 index 0000000000..fa23e206f0 --- /dev/null +++ b/packages/adapter-utils/src/published-exports.test.ts @@ -0,0 +1,38 @@ +import { readFileSync } from "node:fs"; +import { fileURLToPath } from "node:url"; +import { describe, expect, it } from "vitest"; + +// This test pins the published subpath surface of `package.json`. +// +// `@paperclipai/server` is itself a published npm package. Its published +// build imports `@paperclipai/adapter-utils/duplex-observability` as a real +// npm dependency, not as a workspace link. Node resolves that import against +// `publishConfig.exports`, so the subpath must stay published there. An +// earlier revision of this file denied the subpath with an explicit `null` +// entry; that denial broke module resolution for the published server. The +// subpath now falls through to the wildcard entry, the same as every other +// file in the package. + +interface PackageManifest { + exports: Record; + publishConfig: { + exports: Record; + }; +} + +const manifestPath = fileURLToPath(new URL("../package.json", import.meta.url)); +const manifest = JSON.parse(readFileSync(manifestPath, "utf8")) as PackageManifest; + +describe("publishConfig publishes the duplex observability subpath", () => { + it("does not deny the subpath in publishConfig.exports", () => { + expect(manifest.publishConfig.exports).not.toHaveProperty("./duplex-observability"); + }); + + it("keeps the wildcard entry in publishConfig.exports so the subpath resolves through it", () => { + expect(manifest.publishConfig.exports["./*"]).toBeDefined(); + }); + + it("keeps the top-level wildcard export unchanged", () => { + expect(manifest.exports["./*"]).toBe("./src/*.ts"); + }); +}); diff --git a/packages/adapters/claude-local/src/server/execute.ts b/packages/adapters/claude-local/src/server/execute.ts index f8532e8a50..7d2ca4bdb2 100644 --- a/packages/adapters/claude-local/src/server/execute.ts +++ b/packages/adapters/claude-local/src/server/execute.ts @@ -15,7 +15,7 @@ import { ensureAdapterExecutionTargetCommandResolvable, ensureAdapterExecutionTargetRuntimeCommandInstalled, prepareAdapterExecutionTargetRuntime, - adapterExecutionTargetDuplexTelemetryRecorder, + adapterExecutionTargetDuplexObservabilityRecorder, adapterExecutionTargetEnablesSandboxDuplexBridge, readAdapterExecutionTarget, resolveAdapterExecutionTargetTimeoutSec, @@ -684,7 +684,7 @@ export async function execute(ctx: AdapterExecutionContext): Promise` | One inbound asset stage task inside `stage.sync`. It packs and uploads one managed-home asset. The `` segment is the asset key. | `stage.sync` | -| `stage.project.` | One inbound referenced-project stage task inside `stage.sync`. It uploads one referenced project. The `` segment is the project id. | `stage.sync` | -| `pack` | Host-side workspace tarball build inside the `stage.workspace` task. | `stage.workspace` | -| `bridge.paperclip` | Paperclip bridge start step. | `sandbox.startup` | -| `bridge.process-session` | Process-session bridge start step. | `sandbox.startup` | -| `acp.handshake` | ACP session handshake step. | `sandbox.startup` | -| `sandbox.syncBack` | The settlement sync-back that restores the managed home at teardown. | the active run span | -| `restore.workspace` | One outbound workspace restore task at teardown. It reads the sandbox workspace back and merges it into the host workspace. | `sandbox.syncBack` | -| `restore.asset.` | One outbound asset restore task at teardown. It reads one asset back to its host store. The `` segment is the asset key. | `sandbox.syncBack` | -| `sandbox.agentSession.sendInput` | One outbound ACP message to the agent — the socket handler's one `writeTextFile` exec. | the active run span | -| `sandbox.agentSession.pollOutput` | One 100 ms poll tick — `list`, then `read`+`remove` per file found (`1 + 2n` execs). | the active run span | -| `sandbox.callbackBridge.relayRequest` | One Paperclip-API callback request — read the request, write the response, remove it. | the active run span | -| `sandbox.agentProcess` | The persistent streamed agent process the process-session bridge launches; open until the process settles or the bridge tears down, whichever comes first. | the active run span | -| `sandbox.exec` | One host-to-sandbox execution. | the active step or wrapper span | - -A step span name is the step name. The `sandbox.exec` span parents to the step -span that runs the execution, so each execution nests under its step. Within -`stage.sync`, the host-side sub-steps `snapshot.git` and `snapshot.baseline` open -as child spans of the step, so the host work at the head of the step is -attributed rather than showing as a gap. Each inbound sync operation also opens -its own task span under `stage.sync`: `stage.workspace`, one `stage.asset.` -per asset, and one `stage.project.` per referenced project. The `pack` span -nests under `stage.workspace`, because the host builds the tarball inside that -task. Two concurrent tasks produce overlapping spans. - -The settlement `sandbox.syncBack` span runs at teardown and parents to the run -span. It wraps the managed-home restore. Each outbound restore operation opens -its own task span under `sandbox.syncBack`: `restore.workspace` and one -`restore.asset.` per asset. Two concurrent restore tasks produce overlapping -spans. A run-time -`sandbox.exec` span parents instead to the run-time wrapper span that runs it -(`sandbox.agentSession.sendInput`, `sandbox.agentSession.pollOutput`, -`sandbox.callbackBridge.relayRequest`, or `sandbox.agentProcess`). Each run-time -wrapper span parents to the live run span (`agent.turn` during the turn, -`task.run` otherwise). With no active trace context the exec span opens -unparented. - -`sandbox.agentProcess` wraps the persistent streamed agent process. The -process-session bridge launches it during `bridge.process-session`, so it opens -under `task.run` — no turn has started yet. It therefore overlaps the sibling -`agent.turn` rather than nesting under it or dangling off the short-lived bring-up -step. The span ends when the process settles or when the bridge tears down, -whichever comes first. The bridge tears down before the run root span ends, so -the span never outlives `task.run` even when the process lingers past teardown -(the sandbox `execute` has no cancel, so a lingering process cannot be forced to -resolve). - -The root span sets the error status when the bring-up fails. Each step span sets -the error status when its step fails. The `sandbox.exec` span sets the error -status when the exit code is non-zero or the execution throws. - -### Outcome values - -The `paperclip.sandbox.startup.outcome` attribute uses a closed value set: - -- `ok` — the step or the execution settled with a success result. -- `skipped` — a warm cache skipped the step; the step ran no work. -- `failed` — the step or the execution threw, or the exit code was non-zero. - -### Root span attributes - -The `sandbox.startup` root span uses this closed attribute allowlist. - -| Attribute | Type | Optional | Meaning | -| --- | --- | --- | --- | -| `paperclip.sandbox.startup.root.wall_ms` | number | no | The root-span wall time of the whole bring-up. | -| `paperclip.sandbox.startup.root.work_ms` | number | no | The sum of the step wall times. | -| `paperclip.sandbox.startup.root.diff_ms` | number | no | `work_ms − wall_ms`; the overlap the parallel steps saved. | -| `paperclip.sandbox.startup.provider` | string | yes | The normalized provider family. | -| `paperclip.sandbox.startup.cold_start` | boolean | yes | Whether the bring-up is a cold start. | -| `paperclip.sandbox.startup.region` | string | yes | The clamped region label. | -| `paperclip.sandbox.startup.image_id` | string | yes | The hashed image id. | -| `paperclip.sandbox.startup.sandbox_id` | string | yes | The hashed sandbox id. | -| `paperclip.sandbox.startup.lease_id` | string | yes | The hashed lease id. | - -### Step span attributes - -Each bring-up step span uses this closed attribute allowlist. The step name -rides the span name, so no `step` attribute repeats it. - -| Attribute | Type | Optional | Meaning | -| --- | --- | --- | --- | -| `paperclip.sandbox.startup.step.wall_ms` | number | no | The wall time of the step. | -| `paperclip.sandbox.startup.outcome` | string | no | The step outcome (`ok`, `skipped`, or `failed`). | -| `paperclip.sandbox.startup.provider` | string | yes | The normalized provider family. | -| `paperclip.sandbox.startup.batch` | string | yes | A shared tag that marks two parallel steps as one batch. | -| `paperclip.sandbox.startup.handshake.create_runtime.wall_ms` | number | yes | The create-runtime sub-time of the `acp.handshake` step. | -| `paperclip.sandbox.startup.handshake.ensure_session.wall_ms` | number | yes | The ensure-session sub-time of the `acp.handshake` step. | - -The round-trip count and the provider durations no longer ride a step span. The -per-execution `sandbox.exec` child spans carry that detail. - -### `sandbox.exec` span attributes - -The `sandbox.exec` span uses this closed attribute allowlist. Paperclip omits a -numeric attribute when the provider does not report the value. - -| Attribute | Type | Optional | Meaning | -| --- | --- | --- | --- | -| `paperclip.sandbox.startup.provider` | string | no | The normalized provider family. | -| `paperclip.sandbox.startup.exec.command` | string | no | The clamped `argv[0]` command label. | -| `paperclip.sandbox.startup.exec.exit_code` | number | yes | The numeric process exit code. | -| `paperclip.sandbox.startup.exec.wall_ms` | number | no | The host-measured wall time of the execution. | -| `paperclip.sandbox.startup.exec.wait_before_ms` | number | yes | The provider handle-fetch wait before the execution ran. | -| `paperclip.sandbox.startup.exec.sandbox_ms` | number | yes | The in-sandbox run time of the execution. | -| `paperclip.sandbox.startup.exec.network_ms` | number | yes | The transport time the host adds; `wall_ms − wait_before_ms − sandbox_ms`. | -| `paperclip.sandbox.startup.exec.critical_path` | boolean | no | Whether the execution sits on the startup critical path. | -| `paperclip.sandbox.startup.exec.cache_hit` | boolean | yes | Whether the provider served the sandbox handle from its warm cache. | -| `paperclip.sandbox.startup.outcome` | string | no | The execution outcome (`ok` or `failed`). | - -The plugin decides the cache hit at the sandbox-handle lookup. The span no -longer infers a cache hit from `wait_before_ms == 0`. Paperclip omits the -`cache_hit` attribute when the provider does not report the value. - -To add a span attribute, extend the `SANDBOX_STARTUP_SPAN_ATTRS` allowlist in -the code first. Keep the attribute low-cardinality and free of user content. - -### Provider spans - -A sandbox provider plugin also opens spans for its own sync steps. These spans -use the `sandbox.daytona.` name prefix. They share the -`paperclip.sandbox.startup.` attribute prefix and obey the same opt-in and -no-user-content rules as the startup spans above. - -The plugin worker runs in a separate process from the host. So the host treats -every field of a worker-sent span as untrusted input. The host re-clamps the -span name and every attribute at one boundary, the `span.record` host handler, -before it records the span. - -| Span | Scope | Parent | -| --- | --- | --- | -| `sandbox.daytona.pack` | The host-local pack step that builds the upload tarball. It makes no sandbox round trip. | the active startup step span | -| `sandbox.daytona.transfer` | The transfer step: an upload to the sandbox (inbound) or a download from the sandbox (outbound). The `paperclip.sandbox.startup.transfer.direction` attribute records the direction. | the active sync task span (`stage.*` inbound, `restore.*` under `sandbox.syncBack` outbound) | -| `sandbox.daytona.ensureDirectory` | The `mkdir -p` step that ensures a directory exists before a write. | the active startup step span | -| `sandbox.daytona.checkSymlinkEscape` | The re-check step that a path resolves inside the workspace root before use. | the active startup step span | -| `sandbox.daytona.promote` | The atomic move of a staged temp onto its target via a pinned dir handle. | the active startup step span | -| `sandbox.daytona.extractTarball` | The one round trip that re-checks the path, runs `tar -xf`, and removes the scratch tarball. | the active startup step span | -| `sandbox.daytona.postUploadCommand` | One caller-supplied post-upload command. | the active startup step span | -| `sandbox.daytona.session.open` | The create of the one persistent session for a lease, on the first in-run command. | the active run span | -| `sandbox.daytona.session.close` | The delete of that persistent session on lease release. | the active run span | -| `sandbox.daytona.other` | Any span name outside the known set. | the active startup step span | - -The host clamps the span name to the closed set of leaf names above (`pack`, -`transfer`, `ensureDirectory`, `checkSymlinkEscape`, `promote`, `extractTarball`, -`postUploadCommand`, `session.open`, and `session.close`). The host maps a known -name to `sandbox.daytona.`. The host maps any other value to -`sandbox.daytona.other`, so a span name never carries free-form data. Only the -daytona provider emits these spans today, so the segment is the literal -`daytona`. - -The `sandbox.daytona.*` spans use this closed attribute allowlist. The host -drops every other key, so a command, an argument, a path, an id, a standard -output, or a standard error never rides a provider span. The host records only -the attributes that the producer sends for one span. - -| Attribute | Type | Optional | Meaning | -| --- | --- | --- | --- | -| `paperclip.sandbox.startup.provider` | string | no | The normalized provider family. | -| `paperclip.sandbox.startup.outcome` | string | yes | The step outcome (`ok`, `skipped`, or `failed`). | -| `paperclip.sandbox.startup.pack.wall_ms` | number | yes | The host-local wall time of the pack step. It rides the `sandbox.daytona.pack` span. | -| `paperclip.sandbox.startup.transfer.wall_ms` | number | yes | The wall time of the transfer step. It rides the `sandbox.daytona.transfer` span. | -| `paperclip.sandbox.startup.transfer.guard.count` | number | yes | The number of serial guard round trips before one transfer. It rides the `sandbox.daytona.transfer` span. | -| `paperclip.sandbox.startup.transfer.direction` | string | yes | The transfer direction (`inbound` or `outbound`). It rides the `sandbox.daytona.transfer` span. | - -The `span.record` host handler enforces the allowlist. It re-maps `provider` -through the provider-family normalizer. It keeps `outcome` only when the value -is `ok`, `skipped`, or `failed`. It keeps `transfer.direction` only when the -value is `inbound` or `outbound`. It keeps a numeric attribute only when the -value is a finite number. It drops a status message and keeps only the numeric -status code. The handler never throws, because observability must not change the -sync control flow. - -The `span.record` host method needs the `environment.drivers.register` -capability. So only a plugin that registers an environment driver may emit a -provider span. The capability gate rejects a provider span from any other -plugin. - -The host parents each provider span to the active sync task span. An inbound -transfer runs inside a `stage.*` task span, so its provider spans parent there. -An outbound transfer runs inside a `restore.*` task span under `sandbox.syncBack` -at teardown, so its provider spans parent there. The host mints a W3C -`traceparent` from the active task span and passes it to the plugin worker on the -per-call invocation channel. The teardown restore runs inside the run-parented -`sandbox.syncBack` span, so the host mints a `traceparent` for an outbound -provider span the same way it does for an inbound one. The worker tags its span with the -`traceparent` and treats the value as opaque. The worker never derives the -parent from it. The host recovers the `traceparent` from its own invocation -record, so a worker can never forge a parent. The host validates the -`traceparent` and rejects a missing or malformed value. With no active host -trace context the worker sends no span, so the whole provider-span path is a -no-op. - -## Sandbox Duplex Transport Telemetry - -Paperclip opens a fixed observability surface for the sandbox duplex transport. -This surface is separate from the first-party events and from the startup spans -above. The generated telemetry contract does not cover it, so this section is its -canonical contract. The code owner is -`packages/adapter-utils/src/duplex-telemetry.ts`. That module holds each name and -each enum value as a literal constant, so the surface never drifts. - -The surface is opt-in. The host injects a recorder that binds the span to the -OTel tracer, the counter to the guarded counter store in -`server/src/services/tool-runtime-metrics.ts`, and the event to the run-events -bridge. The default recorder is a no-op, so the whole surface stays inert until -the host binds a real recorder. Every recorder call sits inside an error swallow, -so a telemetry failure never breaks the request path. - -The surface carries no user content. No route, no query, no request body, no -token, and no raw identifier rides a span, a counter, or an event. Each record -carries only the closed dimension keys below and, for the request span, a -latency. The `provider` dimension carries only the allowlisted public value -`daytona`. Any other plugin key maps to `other` before the record reaches a sink, -so a raw plugin key never reaches a span attribute, a counter label, or an event -field. - -### Spans - -| Span | Scope | Latency | -| --- | --- | --- | -| `sandbox.duplex.channel_open` | One duplex channel-open attempt. The `outcome` dimension is `ok` when the channel opened and readiness passed, or `error` when the open or readiness failed. | none | -| `sandbox.duplex.request` | One duplex request the broker forwarded to the host. | The request latency in milliseconds. | - -### Event - -| Event | Scope | -| --- | --- | -| `sandbox.duplex.transport` | The host emits it at each transport boundary: a ready duplex channel, a fallback to the file bridge, and a terminal channel loss. Its dimensions record the boundary. | - -### Counters - -| Counter | Scope | -| --- | --- | -| `sandbox_duplex_channel_open_total` | One successful duplex channel open. | -| `sandbox_duplex_fallback_total` | One fallback to the file bridge. The `fallback_reason` dimension records the cause. | -| `sandbox_duplex_loss_total` | One terminal duplex channel loss. The `loss_class` dimension records the phase. | -| `sandbox_duplex_session_leak_total` | One leaked provider session at teardown. | - -### Aggregate byte ledger metrics - -The host aggregate byte ledger owns one process-scoped gauge and two -process-scoped counters. The ledger bounds the retained bytes across every live -duplex route in one process. It sets the gauge on each reserve and each release. -It increments a counter on a rejected reservation and on an accounting defect. -These records carry no dimension label. The guarded counter store keys each -counter on `(companyId, metric)`, and the gauge reports one process value, so no -dynamic dimension rides them. The code owner is -`packages/adapter-utils/src/duplex-aggregate-byte-ledger.ts`, and the metric -names are literal constants in `duplex-telemetry.ts`. - -| Metric | Type | Scope | -| --- | --- | --- | -| `sandbox_duplex_aggregate_bytes_in_use` | gauge | The aggregate retained bytes across every live duplex route. The ledger sets it on each reserve and each release. | -| `sandbox_duplex_aggregate_byte_reservation_rejections_total` | counter | One rejected aggregate byte reservation. The ledger increments it when a reservation would pass the aggregate ceiling. | -| `sandbox_duplex_aggregate_byte_accounting_underflow_total` | counter | One aggregate byte accounting defect. The ledger increments it on a double release or on a transfer of a token it does not hold. | - -### Dimension keys - -Counters carry no dimension labels. The guarded counter store keys each counter -on `(companyId, metric)` with no label column, so the `fallback_reason` and -`loss_class` values fold into the counter metric name instead. The full closed -dimension set below rides only the spans and the `sandbox.duplex.transport` -event, which use only these closed keys. A test asserts the exact set, so a new -key never reaches a sink by accident. - -| Key | Type | Optional | Value set | -| --- | --- | --- | --- | -| `provider` | string | no | `daytona`, or `other` for any other plugin key. | -| `transport` | string | no | `duplex` or `file`. A fallback record uses `file`; every other record uses `duplex`. | -| `outcome` | string | yes | `ok` or `error`. | -| `fallback_reason` | string | yes | `gate_off`, `capability_absent`, `open_failed`, `ready_invalid`, `ready_nonce_mismatch`, `ready_timeout`, or `contaminated`. It rides only a fallback record. | -| `loss_class` | string | yes | `pre_dispatch` or `post_dispatch`, relative to the first request dispatch. It rides only a loss record. | -| `loss_reason` | string | yes | `stdin_eof`, `provider_exit`, `heartbeat_timeout`, `rpc_failure`, `write_error`, `transport_closed`, or `other`. The host maps every loss cause to one of these values, so no raw provider text reaches a sink. `write_error` marks a rejected host-to-sandbox write. `transport_closed` marks a reason-less provider transport close with no exit data. It rides only a loss record. | - -To add a name or an enum value, extend the literal constant in -`duplex-telemetry.ts` first, then update the test that asserts the closed set. -Keep every dimension low-cardinality and free of user content. - -## Sandbox Startup Run-Log Event - -Paperclip writes one `run.startup.step` event to the run log for each bring-up -step. This event is a run-log record, not a first-party telemetry event. The -generated telemetry contract does not cover it, so this section is its canonical -contract. - -The event payload carries only three fields. - -| Field | Type | Meaning | -| --- | --- | --- | -| `step` | string | The bring-up step name, for example `stage.sync`. | -| `durationMs` | number | The wall time of the step. A skipped step reports `0`. | -| `outcome` | string | The step outcome (`ok`, `skipped`, or `failed`). | - -The event no longer carries the per-step round-trip count or the provider -duration fields. It dropped `roundTrips`, `providerExecMs`, `providerGetMs`, -`createRuntimeMs`, and `ensureSessionMs`. The startup spans in the section above -carry that detail now. The `sandbox.exec` child spans hold the round-trip and -provider durations. The `acp.handshake` step span holds the create-runtime and -ensure-session sub-times. - -To read the detailed timing, use the startup spans. The spans need an OTLP -endpoint. A run with no endpoint keeps only the three run-log fields above. - -## Run Phase Timing Run-Log Event - -Paperclip writes one `run.phase.timing` event to the run log for each -run-lifecycle phase. This event is a run-log record, not a first-party telemetry -event. The generated telemetry contract does not cover it, so this section is its -canonical contract. The producer is `emitRunPhaseTiming` in -`packages/adapter-utils/src/acpx-engine/startup-timing.ts`. - -The event payload carries only three fields. - -| Field | Type | Meaning | -| --- | --- | --- | -| `phase` | string | The run-lifecycle phase name from the closed allowlist below. | -| `durationMs` | number | The wall time of the phase. A negative or a non-finite value clamps to `0`. | -| `outcome` | string | The phase outcome (`ok` or `failed`). | - -The `phase` field is one member of a closed, low-cardinality allowlist. The -producer drops any event whose phase name is outside this allowlist, so a -free-form label never reaches the run log. The allowlist has twelve phase names. - -| Phase | Meaning | -| --- | --- | -| `place_workspace` | Place the run workspace. | -| `start_transport` | Start the agent transport. | -| `create_runtime` | Create the agent runtime. | -| `ensure_session` | Ensure the agent session exists. | -| `configure_session` | Configure the agent session. | -| `prepare_turn` | Prepare the turn. | -| `turn` | Run the turn. | -| `end_session` | End the agent session. | -| `settle_reuse` | Settle the session for reuse. | -| `stop_transport` | Stop the agent transport. | -| `sync_back` | Sync the workspace back. | -| `release_staging_lease` | Release the staging lease. | - -The payload never carries a command, an argument, a path, an environment value, -or a raw identifier. The event rides the `ctx.onEvent` run-event bridge and is -observability-only. It needs no OTLP endpoint. +- [Observability](../../../../doc/observability.md) — the OpenTelemetry trace + path, the sandbox startup trace spans, and the sandbox duplex transport + instrumentation. +- [Run-Log Events](../../../../doc/run-log-events.md) — events written to the + local `heartbeat_run_events` table. ## Dimension Values diff --git a/scripts/general-server-shard-durations.json b/scripts/general-server-shard-durations.json index 4e3a786f8a..9f8ff9e4fe 100644 --- a/scripts/general-server-shard-durations.json +++ b/scripts/general-server-shard-durations.json @@ -99,7 +99,7 @@ "server/src/__tests__/documents-service.test.ts": 4497, "server/src/__tests__/documents.test.ts": 1104, "server/src/__tests__/duplex-aggregate-ceiling-env.test.ts": 210, - "server/src/__tests__/duplex-telemetry-recorder.test.ts": 207, + "server/src/__tests__/duplex-observability-recorder.test.ts": 207, "server/src/__tests__/effective-run-config-fingerprints.test.ts": 219, "server/src/__tests__/embedded-postgres-supervisor.test.ts": 206, "server/src/__tests__/environment-capability-contract.test.ts": 1735, diff --git a/server/src/__tests__/duplex-telemetry-recorder.test.ts b/server/src/__tests__/duplex-observability-recorder.test.ts similarity index 87% rename from server/src/__tests__/duplex-telemetry-recorder.test.ts rename to server/src/__tests__/duplex-observability-recorder.test.ts index 4edddcedce..50c10c9a4a 100644 --- a/server/src/__tests__/duplex-telemetry-recorder.test.ts +++ b/server/src/__tests__/duplex-observability-recorder.test.ts @@ -7,25 +7,25 @@ import { DUPLEX_SPAN_CHANNEL_OPEN, DUPLEX_SPAN_REQUEST, DUPLEX_TRANSPORT_EVENT, -} from "@paperclipai/adapter-utils/duplex-telemetry"; +} from "@paperclipai/adapter-utils/duplex-observability"; import { - createHostDuplexTelemetryRecorder, + createHostDuplexObservabilityRecorder, foldDuplexCounterMetric, - type DuplexTelemetrySpan, - type DuplexTelemetryTracer, -} from "../services/duplex-telemetry-recorder.js"; + type DuplexObservabilitySpan, + type DuplexObservabilityTracer, +} from "../services/duplex-observability-recorder.js"; // A recording tracer that captures each span's name, attributes, and end time. function createRecordingTracer(): { - tracer: DuplexTelemetryTracer; + tracer: DuplexObservabilityTracer; spans: Array<{ name: string; startTime?: number; attributes: Record; endTime?: number }>; } { const spans: Array<{ name: string; startTime?: number; attributes: Record; endTime?: number }> = []; - const tracer: DuplexTelemetryTracer = { + const tracer: DuplexObservabilityTracer = { startSpan(name, options) { const record = { name, startTime: options?.startTime, attributes: {} as Record, endTime: undefined as number | undefined }; spans.push(record); - const span: DuplexTelemetrySpan = { + const span: DuplexObservabilitySpan = { setAttribute(key, value) { record.attributes[key] = value; }, @@ -39,10 +39,10 @@ function createRecordingTracer(): { return { tracer, spans }; } -describe("createHostDuplexTelemetryRecorder", () => { +describe("createHostDuplexObservabilityRecorder", () => { it("records the channel-open span with only the closed dimension keys", () => { const { tracer, spans } = createRecordingTracer(); - const recorder = createHostDuplexTelemetryRecorder({ + const recorder = createHostDuplexObservabilityRecorder({ tracer, incrementCounter: () => {}, emitTransportEvent: () => {}, @@ -64,7 +64,7 @@ describe("createHostDuplexTelemetryRecorder", () => { it("makes the request span duration equal the measured latency", () => { const { tracer, spans } = createRecordingTracer(); - const recorder = createHostDuplexTelemetryRecorder({ + const recorder = createHostDuplexObservabilityRecorder({ tracer, incrementCounter: () => {}, emitTransportEvent: () => {}, @@ -104,7 +104,7 @@ describe("createHostDuplexTelemetryRecorder", () => { it("forwards the counter through the guarded sink with the folded metric", () => { const metrics: string[] = []; - const recorder = createHostDuplexTelemetryRecorder({ + const recorder = createHostDuplexObservabilityRecorder({ tracer: createRecordingTracer().tracer, incrementCounter: (metric) => metrics.push(metric), emitTransportEvent: () => {}, @@ -120,7 +120,7 @@ describe("createHostDuplexTelemetryRecorder", () => { it("forwards the transport event with its name and dimensions", () => { const events: Array<{ name: string; dimensions: Record }> = []; - const recorder = createHostDuplexTelemetryRecorder({ + const recorder = createHostDuplexObservabilityRecorder({ tracer: createRecordingTracer().tracer, incrementCounter: () => {}, emitTransportEvent: (event) => events.push(event), diff --git a/server/src/index.ts b/server/src/index.ts index 058348ddc3..aa414603dc 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -89,7 +89,7 @@ import { type DuplexAggregateByteLedgerMetricSink, } from "@paperclipai/adapter-utils/duplex-aggregate-byte-ledger"; import { resolveDuplexAggregateCeilingBytesFromEnv } from "./duplex-aggregate-ceiling-env.js"; -import { DUPLEX_COUNTER_AGGREGATE_BYTE_ACCOUNTING_UNDERFLOW_TOTAL } from "@paperclipai/adapter-utils/duplex-telemetry"; +import { DUPLEX_COUNTER_AGGREGATE_BYTE_ACCOUNTING_UNDERFLOW_TOTAL } from "@paperclipai/adapter-utils/duplex-observability"; import { createStorageServiceFromConfig } from "./storage/index.js"; import { printStartupBanner } from "./startup-banner.js"; import { getBoardClaimWarningUrl, initializeBoardClaimChallenge } from "./board-claim.js"; diff --git a/server/src/services/duplex-telemetry-recorder.ts b/server/src/services/duplex-observability-recorder.ts similarity index 75% rename from server/src/services/duplex-telemetry-recorder.ts rename to server/src/services/duplex-observability-recorder.ts index 8a4192dae6..0962c52585 100644 --- a/server/src/services/duplex-telemetry-recorder.ts +++ b/server/src/services/duplex-observability-recorder.ts @@ -1,12 +1,12 @@ import { DUPLEX_DIMENSION_KEYS, DUPLEX_SPAN_REQUEST, - type DuplexTelemetryCounterRecord, - type DuplexTelemetryDimensions, - type DuplexTelemetryEventRecord, - type DuplexTelemetryRecorder, - type DuplexTelemetrySpanRecord, -} from "@paperclipai/adapter-utils/duplex-telemetry"; + type DuplexObservabilityCounterRecord, + type DuplexObservabilityDimensions, + type DuplexObservabilityEventRecord, + type DuplexObservabilityRecorder, + type DuplexObservabilitySpanRecord, +} from "@paperclipai/adapter-utils/duplex-observability"; /** * The host binding for the fixed duplex telemetry surface. This module maps each @@ -15,7 +15,7 @@ import { * to the run-event path. The mapping is the single boundary, so the closed * dimension keys and the counter folding stay in one place. * - * The recorder never throws. The facade in `duplex-telemetry.ts` wraps each + * The recorder never throws. The facade in `duplex-observability.ts` wraps each * synchronous call in a swallow, but an async sink can still reject after the * call returns. Each async sink here runs fire-and-forget with its own catch, so * a sink failure never breaks the request path. @@ -23,7 +23,7 @@ import { /** The minimal span the recorder opens. A real OTel span satisfies it; the * no-op tracer's span satisfies it too. */ -export interface DuplexTelemetrySpan { +export interface DuplexObservabilitySpan { setAttribute(key: string, value: string | number | boolean): void; end(endTime?: number): void; } @@ -32,13 +32,13 @@ export interface DuplexTelemetrySpan { * real or a no-op implementation that satisfies it. The optional second argument * carries an explicit start time, so the request span duration equals the * measured latency. */ -export interface DuplexTelemetryTracer { - startSpan(name: string, options?: { startTime?: number }): DuplexTelemetrySpan; +export interface DuplexObservabilityTracer { + startSpan(name: string, options?: { startTime?: number }): DuplexObservabilitySpan; } -export interface HostDuplexTelemetryRecorderInput { +export interface HostDuplexObservabilityRecorderInput { /** The OTel tracer for the two duplex spans. */ - tracer: DuplexTelemetryTracer; + tracer: DuplexObservabilityTracer; /** * Increment one guarded host counter by its metric name. The caller binds it * to `incrementToolRuntimeMetricCounter` with the company id, inside a swallow. @@ -48,7 +48,7 @@ export interface HostDuplexTelemetryRecorderInput { * Emit one transport event to the run-event path. The caller binds it to the * run-events bridge, inside a swallow. */ - emitTransportEvent(event: { name: string; dimensions: DuplexTelemetryDimensions }): void; + emitTransportEvent(event: { name: string; dimensions: DuplexObservabilityDimensions }): void; /** The wall clock. The default is `Date.now`. Tests inject a fixed clock. */ now?: () => number; } @@ -60,7 +60,7 @@ export interface HostDuplexTelemetryRecorderInput { * breakdown. Both dimension sets are closed and low-cardinality, so the metric * cardinality stays bounded. A record with neither dimension uses the base name. */ -export function foldDuplexCounterMetric(record: DuplexTelemetryCounterRecord): string { +export function foldDuplexCounterMetric(record: DuplexObservabilityCounterRecord): string { if (record.dimensions.fallback_reason) { return `${record.metric}.${record.dimensions.fallback_reason}`; } @@ -75,12 +75,12 @@ export function foldDuplexCounterMetric(record: DuplexTelemetryCounterRecord): s * already-mapped dimensions, so a raw provider key never reaches a sink. Every * span attribute uses only the closed dimension keys. */ -export function createHostDuplexTelemetryRecorder( - input: HostDuplexTelemetryRecorderInput, -): DuplexTelemetryRecorder { +export function createHostDuplexObservabilityRecorder( + input: HostDuplexObservabilityRecorderInput, +): DuplexObservabilityRecorder { const now = input.now ?? Date.now; - const setDimensionAttributes = (span: DuplexTelemetrySpan, dimensions: DuplexTelemetryDimensions): void => { + const setDimensionAttributes = (span: DuplexObservabilitySpan, dimensions: DuplexObservabilityDimensions): void => { for (const key of DUPLEX_DIMENSION_KEYS) { const value = dimensions[key]; if (typeof value === "string") { @@ -90,7 +90,7 @@ export function createHostDuplexTelemetryRecorder( }; return { - recordSpan(record: DuplexTelemetrySpanRecord): void { + recordSpan(record: DuplexObservabilitySpanRecord): void { // The request span carries a latency, so start it in the past and end it // now, so the span duration equals the measured latency. The channel-open // span carries no latency, so it opens and ends at the same instant. @@ -103,10 +103,10 @@ export function createHostDuplexTelemetryRecorder( setDimensionAttributes(span, record.dimensions); span.end(end); }, - incrementCounter(record: DuplexTelemetryCounterRecord): void { + incrementCounter(record: DuplexObservabilityCounterRecord): void { input.incrementCounter(foldDuplexCounterMetric(record)); }, - emitEvent(record: DuplexTelemetryEventRecord): void { + emitEvent(record: DuplexObservabilityEventRecord): void { input.emitTransportEvent({ name: record.name, dimensions: record.dimensions }); }, }; diff --git a/server/src/services/environment-execution-target.ts b/server/src/services/environment-execution-target.ts index 635278bc1c..9730708f35 100644 --- a/server/src/services/environment-execution-target.ts +++ b/server/src/services/environment-execution-target.ts @@ -5,7 +5,7 @@ import { adapterExecutionTargetToRemoteSpec, type AdapterExecutionTarget, } from "@paperclipai/adapter-utils/execution-target"; -import type { DuplexTelemetryRecorder } from "@paperclipai/adapter-utils/duplex-telemetry"; +import type { DuplexObservabilityRecorder } from "@paperclipai/adapter-utils/duplex-observability"; import type { DuplexAggregateByteLedger } from "@paperclipai/adapter-utils/duplex-aggregate-byte-ledger"; import { clampSpanLabel, @@ -203,11 +203,11 @@ export async function resolveEnvironmentExecutionTarget(input: { // gated server tracer, which is a no-op when tracing is off. Tests inject a // recording tracer. tracer?: ExecTracer; - // The host duplex telemetry recorder. The seam stamps it onto the sandbox + // The host duplex observability recorder. The seam stamps it onto the sandbox // target next to the runner, so the live object stays on the host and never // enters the sandbox environment. Absent keeps the safe no-op default in the // bridge, so the surface stays inert until the host injects a real recorder. - duplexTelemetryRecorder?: DuplexTelemetryRecorder | null; + duplexObservabilityRecorder?: DuplexObservabilityRecorder | null; // The process-owned aggregate byte ledger. The seam stamps it onto the sandbox // target next to the runner, so the live object stays on the host and never // enters the sandbox environment. Absent keeps the bridge inert for this seam. @@ -336,10 +336,10 @@ export async function resolveEnvironmentExecutionTarget(input: { shellCommand, remoteCwd, enableSandboxDuplexBridge, - // Attach the host duplex telemetry recorder next to the runner. The bridge + // Attach the host duplex observability recorder next to the runner. The bridge // binds it to the fixed observability surface. Absent keeps the no-op // default, so the surface stays inert on a run with no injected recorder. - duplexTelemetryRecorder: input.duplexTelemetryRecorder ?? null, + duplexObservabilityRecorder: input.duplexObservabilityRecorder ?? null, // Attach the process-owned aggregate byte ledger next to the runner. The // bridge passes it to the broker, the decoder, and the response-body reader. // Absent keeps the bridge inert for this seam. diff --git a/server/src/services/environment-run-orchestrator.ts b/server/src/services/environment-run-orchestrator.ts index 8ef5aa9d03..9cf984f3b6 100644 --- a/server/src/services/environment-run-orchestrator.ts +++ b/server/src/services/environment-run-orchestrator.ts @@ -43,7 +43,7 @@ import { type AdapterRemoteExecutionSpec, type AdapterWorkspaceRealization, } from "@paperclipai/adapter-utils/execution-target"; -import type { DuplexTelemetryRecorder } from "@paperclipai/adapter-utils/duplex-telemetry"; +import type { DuplexObservabilityRecorder } from "@paperclipai/adapter-utils/duplex-observability"; import type { DuplexAggregateByteLedger } from "@paperclipai/adapter-utils/duplex-aggregate-byte-ledger"; import { buildWorkspaceRealizationRequest } from "./workspace-realization.js"; import { executionWorkspaceService } from "./execution-workspaces.js"; @@ -359,11 +359,11 @@ export function environmentRunOrchestrator( effectiveExecutionWorkspaceMode: string | null; persistedExecutionWorkspace: ExecutionWorkspace | null; /** - * The host duplex telemetry recorder for this run. The orchestrator threads + * The host duplex observability recorder for this run. The orchestrator threads * it to `resolveEnvironmentExecutionTarget`, which stamps it on the sandbox * target. Absent keeps the safe no-op default in the bridge. */ - duplexTelemetryRecorder?: DuplexTelemetryRecorder | null; + duplexObservabilityRecorder?: DuplexObservabilityRecorder | null; }): Promise { const { environment, @@ -528,7 +528,7 @@ export function environmentRunOrchestrator( leaseMetadata: (lease.metadata as Record | null) ?? null, lease, environmentRuntime, - duplexTelemetryRecorder: input.duplexTelemetryRecorder ?? null, + duplexObservabilityRecorder: input.duplexObservabilityRecorder ?? null, duplexAggregateByteLedger: options.duplexAggregateByteLedger ?? null, }); const realizationMode = workspaceRealization.mode === "in_place" ? "in_place" : "copy"; diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index b85486061f..c2c0d18d01 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -71,7 +71,7 @@ import { } from "@paperclipai/db"; import { conflict, HttpError, notFound } from "../errors.js"; import { getStartupTraceContext, getStartupTracer } from "../instrumentation.js"; -import { createHostDuplexTelemetryRecorder } from "./duplex-telemetry-recorder.js"; +import { createHostDuplexObservabilityRecorder } from "./duplex-observability-recorder.js"; import type { DuplexAggregateByteLedger } from "@paperclipai/adapter-utils/duplex-aggregate-byte-ledger"; import { incrementToolRuntimeMetricCounter } from "./tool-runtime-metrics.js"; import { logger } from "../middleware/logger.js"; @@ -15358,13 +15358,13 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {}) lease: acquiredEnvironment.lease, leaseContext: acquiredEnvironment.leaseContext, }; - // The host duplex telemetry recorder for this run. It binds the fixed duplex + // The host duplex observability recorder for this run. It binds the fixed duplex // observability surface to real sinks: the spans to the OTel tracer, the // guarded counters to the tool-runtime metric store, and the transport event // to the run-event path. Each sink runs guarded and fire-and-forget, so a // telemetry failure never breaks the run. The orchestrator stamps it on the // sandbox target; a non-duplex run keeps the safe no-op default in the bridge. - const duplexTelemetryRecorder = createHostDuplexTelemetryRecorder({ + const duplexObservabilityRecorder = createHostDuplexObservabilityRecorder({ tracer: getStartupTracer(), incrementCounter: (metric) => { void incrementToolRuntimeMetricCounter(db, { @@ -15395,7 +15395,7 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {}) executionWorkspace, effectiveExecutionWorkspaceMode, persistedExecutionWorkspace, - duplexTelemetryRecorder, + duplexObservabilityRecorder, }); activeEnvironmentLease = { ...activeEnvironmentLease,