RomanNum3ral
/
paperclip
mirror of https://github.com/paperclipai/paperclip.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
paperclip/doc/observability.md

847 lines
54 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Observability
This document is the Observability contract. It covers the OpenTelemetry
trace path, the opt-in Sentry error-monitoring path, and two local
instrumentation contracts; see the
[Telemetry Data Contract](../packages/shared/src/telemetry/README.md) for the
separate first-party event system.
Paperclip ships with **opt-in** OpenTelemetry auto-instrumentation for the
server process. When activated it produces **traces only** — no metrics and no
logs are exported by this integration.
`@opentelemetry/api` is a normal dependency of `@paperclipai/server`. Every
install includes it. It stays a no-op interface until an SDK registers a
provider, so it exports no telemetry by itself.
The SDK, the auto-instrumentation bundle, and the resources and
semantic-conventions helpers are *optional peer dependencies*: they are not in
the default lockfile, and the server loads them dynamically only when an
operator turns the feature on. The three exporters below are mutually
alternative peer dependencies — install exactly **one**, matching
`OTEL_EXPORTER_OTLP_PROTOCOL`.
When `OTEL_EXPORTER_OTLP_ENDPOINT` is unset, none of the `@opentelemetry/*` SDK
packages are imported and there is zero runtime overhead.
`server/package.json` declares each optional package at the exact version the
server tests against; install that exact version. Our Dependabot cannot bump
these versions: its npm parser reads only `dependencies`, `devDependencies`,
and `optionalDependencies`, never `peerDependencies`. A peer version here is a
compatibility claim, not an installed version, so raising it is a human
decision. `@opentelemetry/api` is the one OpenTelemetry package Paperclip
maintains as a dependency; once you install the packages below, they become
normal dependencies of **your own** project, and your own Dependabot updates
them.
## Enabling tracing
### 1. Install the OTel peer dependencies
Install the SDK, the auto-instrumentations bundle, the resources/semconv
helpers, and **one** exporter matching your chosen OTLP protocol, at the exact
versions below.
Common to every protocol:
```bash
pnpm add \
@opentelemetry/sdk-node@0.221.0 \
@opentelemetry/auto-instrumentations-node@0.79.0 \
@opentelemetry/resources@2.10.0 \
@opentelemetry/semantic-conventions@1.43.0
```
Then add the exporter for the protocol you intend to use:
| `OTEL_EXPORTER_OTLP_PROTOCOL` | Exporter package | Version |
| ------------------------------ | ------------------------------------------- | --------- |
| `grpc` (default if unset) | `@opentelemetry/exporter-trace-otlp-grpc` | `0.221.0` |
| `http/protobuf` | `@opentelemetry/exporter-trace-otlp-proto` | `0.221.0` |
| `http/json` | `@opentelemetry/exporter-trace-otlp-http` | `0.221.0` |
For example, for the default gRPC path:
```bash
pnpm add @opentelemetry/exporter-trace-otlp-grpc@0.221.0
```
### 2. Set the environment
Minimal setup:
```bash
# Required — turns the feature on. Point at your collector.
# For grpc this is the gRPC target (typically port 4317). For the HTTP
# protocols give the collector's BASE URL (typically port 4318) — the
# exporter appends /v1/traces itself.
export OTEL_EXPORTER_OTLP_ENDPOINT="http://otel-collector:4317"
# Optional — protocol. Defaults to grpc when unset.
# Valid values: grpc | http/protobuf | http/json
export OTEL_EXPORTER_OTLP_PROTOCOL="grpc"
# Optional — service identity attached to every span.
export OTEL_SERVICE_NAME="paperclip"
export OTEL_SERVICE_VERSION="2026.5.0"
```
### `service.version` resolution order
The `service.version` span attribute reports the commit the running server was
built from. The server resolves it in this order and uses the first source that
returns a value:
1. **The build stamp.** The server `build` script writes the commit SHA into
`dist/build-info.json`. The stamp wins so the reported version tracks the
true built commit and cannot go stale across rebuilds. The build script
reads the commit from `git rev-parse --short HEAD` first. A Docker image
build excludes `.git`, so the build script reads the `PAPERCLIP_BUILD_COMMIT`
environment variable instead. Pass the built commit in that variable so the
image stamp records the true commit.
2. **A runtime `git rev-parse --short HEAD`.** This covers `tsx src/index.ts`
dev mode, where the server runs from the source checkout and writes no
stamp. A failure here is not fatal.
3. **The `OTEL_SERVICE_VERSION` environment variable.** This is the fallback
for a build with no stamp and no reachable git — for example a tarball
build. `OTEL_SERVICE_VERSION` is a Paperclip-specific variable, not an
OpenTelemetry SDK variable, so Paperclip controls this precedence.
4. **`"unknown"`** when no source returns a value.
The server logs the resolved `service.version` once at startup, so an operator
can confirm the value.
If `OTEL_EXPORTER_OTLP_PROTOCOL` is set to an unrecognized value, Paperclip
logs a single warning and falls back to gRPC.
Before it imports any OTel package, the server checks the four common
packages and the selected exporter against the exact versions
`server/package.json` declares. If `OTEL_EXPORTER_OTLP_ENDPOINT` is set and a
package is missing or installed at a different version, the server logs one
diagnostic line on boot and continues without tracing — your server stays up.
## Scope
The OpenTelemetry export carries **traces only**. Metrics and log exporters
are out of scope and intentionally not configured here. Auto-instrumentations
for `fs`, `dns`, and `net` are disabled by default because they are too chatty
for this workload; everything else from
`@opentelemetry/auto-instrumentations-node` is on (HTTP, Express, PG, etc.).
This document also holds three local instrumentation contracts: native runner
traces, sandbox startup traces, and sandbox duplex transport instrumentation.
Those sections follow below.
## Native Runner Trace Spans
Paperclip Runner task runs emit a single foldable OpenTelemetry trace. This is
the native-run trace schema version `2`. `task.run` is the only full-run root;
every other native span carries a real OpenTelemetry parent context rather than
only a descriptive `parentName` field.
The canonical lifecycle is:
```text
task.run
├── heartbeat.queue
├── task.prepare
│ ├── environment.startup
│ │ ├── environment.acquire
│ │ └── environment.workspace.realize
│ ├── skills.prepare
│ ├── heartbeat.prepare_before_environment
│ ├── heartbeat.prepare_after_environment
│ └── native.coordinator.claim
├── native.session.execute
│ ├── runner.session.startup
│ │ ├── runner.transport.connect
│ │ │ └── runner.artifact.prepare
│ │ ├── runner.transport.activation
│ │ ├── runner.transport.ready
│ │ ├── runner.runtime.stage
│ │ │ └── stage.sync
│ │ │ ├── stage.asset.home
│ │ │ │ └── session.checkpoint.restore
│ │ │ ├── stage.asset.runtime_context
│ │ │ └── stage.asset.ca_bundle
│ │ ├── runner.session.bootstrap | runner.session.resume
│ │ └── runner.turn.submit
│ └── agent.turn
│ ├── provider.turn.queue
│ └── provider.time_to_first_agent_event
└── task.settle
├── native.result.finalize
└── session.checkpoint.persist
```
The tree shows stable semantic groups, not an exhaustive leaf list. Existing
artifact discovery and verification, process launch, PRP/websocket, ingress,
sandbox lease, harness-state, provider, duplex, and `sandbox.exec` spans remain
under the closest group. This keeps detailed diagnosis available while letting
a trace UI collapse the run into preparation, runner startup, agent work, and
settlement. A repeated operation creates another span with the same semantic
name; attempts are not encoded into span names.
`runner.session.startup` ends at the first durable `turn.submitted` event. A
fresh provider session records `runner.session.bootstrap`; an exact recovered
session records `runner.session.resume`. `agent.turn` begins at
`turn.submitted` and ends at the provider terminal event. `task.settle` begins
at that terminal event and remains open through finalization and checkpoint
persistence, so settlement work does not appear to outlive its parent.
The active native scope is also published through the existing asynchronous
runtime-parent seam. Provider execution, plugin, websocket/duplex, daemon, and
sandbox spans therefore inherit the correct branch even when their callbacks
run in another service layer. With no active native scope, those existing seams
retain their documented fallback behavior.
The root carries only a hashed run id, runtime label, schema version, wall time,
and outcome. Native child attributes use the bounded
`paperclip.native.span.` prefix and a closed key allowlist; values are limited
to finite numbers, booleans, or short strings. Commands, arguments, environment
values, paths, output, credentials, and raw identifiers are discarded by the
trace helper. `task.run.measured` remains in the local run log for compatibility
but is not exported as a second full-width OTel span.
Persisted `run.performance.span` events retain the v1 run-log schema and include
`traceSchemaVersion: 2` so local tooling can distinguish the hierarchy.
Like every span in this document, native-run spans are opt-in. When
`OTEL_EXPORTER_OTLP_ENDPOINT` is unset, the tracer remains a no-op; the local
run-log copy is unaffected.
## Sentry Error Monitoring
Paperclip ships with **opt-in** Sentry error monitoring for the server
process and the browser app. The operator activates it with two
environment variables: `SENTRY_DSN_FRONTEND` for the browser and
`SENTRY_DSN_BACKEND` for the server. Each variable is optional. A
specific variable always wins for its own component; a legacy variable,
`SENTRY_DSN`, supplies a component that has no specific value set. An
empty string counts as absent for all three variables. The feature uses
built-in Sentry options only. It adds no `beforeSend` hook and no custom
filter code.
The server is inactive when the backend DSN resolves to `null`; then it
imports no Sentry package. The browser is inactive when the front-end DSN
resolves to `null`; then it fetches no Sentry chunk. The two components
resolve their DSN independently, so the operator can activate one
component and leave the other inactive.
### Enabling Sentry
#### 1. Install the Sentry peer dependency
The supported server SDK version is **`@sentry/node@10.71.0`** — the exact
version this feature is audited against (see "Server request data"
below). Install it in the server, the same way you install the
OpenTelemetry packages above. `@sentry/node` is an *optional peer
dependency*: it is not in the default lockfile, and the server loads it
dynamically only when the backend DSN resolves to a value.
`server/package.json` declares this exact version; installing a different
version defeats the audit, so the server checks the installed version
against the declared one at startup and logs one diagnostic instead of
enabling error monitoring on a mismatch (see "Server request data"
below).
```bash
pnpm add @sentry/node@10.71.0
```
**The hosted image variant ships this package pre-installed.** A managed
tenant runs the image built from the Dockerfile's `cloud` target, and that
target installs the declared version of `@sentry/node` at build time. A
managed tenant needs only `SENTRY_DSN_BACKEND` set (or `SENTRY_DSN_FRONTEND`
for the browser); no install step is needed.
A self-hosted operator runs the image built from the `production` target.
That image holds no Sentry package, the same as before this feature
existed. A self-hosted operator who wants server error monitoring still
completes the install step above.
The browser package, `@sentry/browser`, needs no install step. It is
already a development dependency of the `ui` package, pinned to the same
exact version, **`10.71.0`**, so the browser code ships inside every
build at the audited version. A signed-out browser, or a browser with no
DSN, never fetches the Sentry chunk — see "DSN delivery to the browser"
below.
#### 2. Set the environment
```bash
export SENTRY_DSN_FRONTEND="https://<public-key>@<host>/<project-id>"
export SENTRY_DSN_BACKEND="https://<public-key>@<host>/<project-id>"
```
The operator can set either variable alone. The component with no value
set stays inactive.
### Two Sentry projects
The server and the browser report to two separate Sentry projects by
default, one per component. The server reads its DSN,
`SENTRY_DSN_BACKEND`, from the process environment. The browser reads its
DSN, `SENTRY_DSN_FRONTEND`, from the authenticated
`GET /api/auth/get-session` response.
The legacy `SENTRY_DSN` variable still works. When the operator sets only
`SENTRY_DSN`, both components use it, so both report to **one** Sentry
project. In that mode the server prints one warning at start. The warning
names the three variables (`SENTRY_DSN`, `SENTRY_DSN_FRONTEND`,
`SENTRY_DSN_BACKEND`) and prints no DSN value.
To add a DSN for a new component later, add a field to the `SentryDsns`
type, add a variable with the `SENTRY_DSN_` prefix, and resolve it with
the same precedence rule: the specific variable wins, and `SENTRY_DSN`
supplies a component that has no specific value set.
### DSN delivery to the browser
The browser never reads the DSN from a `<meta>` tag or from any other part
of `index.html`. The served `index.html` holds no DSN — it is a static
file, built once and served unchanged to every request.
Instead, the browser receives the front-end DSN inside the authenticated
`GET /api/auth/get-session` response body, in the `sentryDsn` field, next
to the signed-in session and the user profile. The backend DSN stays in
the server process and never reaches the browser. A signed-out browser
calls this route with no board actor, so the route answers 401 and sends
no DSN. A signed-out browser therefore loads no Sentry chunk and sends no
event. These pages run signed out:
- `/auth`
- `/cli-auth/:id`
- `/board-claim/:token`
- `/invite/:token`
**A gap the operator must know:** a browser error that happens before the
session response arrives is not captured. The gate opens only after the
session query resolves.
### Privacy settings
The feature uses built-in Sentry options only.
- `sendDefaultPii` is `false`, on both runtimes.
- `tracesSampleRate` is `0`, on both runtimes. Paperclip sends no
performance trace and no profile.
- There is no `beforeSend` hook and no custom filter, on either runtime.
### Server request data
**A server event carries no request data at all.** It holds no URL, no
method, no header, no cookie, no query string, and no body. This is a
verified result, not the Sentry SDK's documented default. A live test
against the real `@sentry/node@10.71.0` package proves it: it captures an
event from inside a real HTTP request handler and confirms the event
holds no request field (see `server/src/__tests__/sentry.test.ts`, "a
server event captured inside a real HTTP request handler carries no
request field").
The reason is `skipOpenTelemetrySetup: true`. This feature sets that
option so it never fights Paperclip's separate, independently opt-in
OpenTelemetry feature for control of the global tracer. The same option
turns off Sentry's per-request context tracking. Sentry's built-in
`RequestData` integration needs that tracking to find a URL, a method, a
header set, a cookie set, or a query string to attach. `RequestData`
stays in the integration list — the initializer does not remove it — but
it attaches nothing under this configuration.
This holds even when the operator turns on the separate OpenTelemetry
feature too (`OTEL_EXPORTER_OTLP_ENDPOINT` set). A live test with a real
OpenTelemetry SDK, a real HTTP instrumentation package, and a real
async-context manager registered still shows no request field on the
captured event.
A server event carries only the exception, its stack trace, and the
context the other kept default integrations add: the host name, the
runtime version, and the dependency list. See "Default capture set"
below.
### Browser data
The browser sends no page URL, no referrer, no user agent, and no
breadcrumb.
### Fail-open behavior
A failed Sentry import or a failed init never stops the server and never
breaks the browser app. Both runtimes fall through to a single diagnostic
log line and keep running with no error monitoring.
### Default capture set
The lists below name every event and every context field this feature
sends, so an operator can read what the feature does before turning it on.
Each Sentry integration name below is verified against the default
integration list of `@sentry/node@10.71.0` and `@sentry/browser@10.71.0`.
**Server events this feature adds**
- An Express `HttpError` with `status >= 500`.
- Any unknown throw that is not a `ZodError`. It always answers 500.
- A server startup failure.
**Server events the default integrations add**
- `OnUncaughtException` — each uncaught exception on the main thread, at
level `fatal`. The process still exits.
- `OnUnhandledRejection` — each unhandled promise rejection. The mode is
`strict`, so the process exits after the capture.
- `ChildProcess` — one event for each worker-thread `error`.
- `LinkedErrors` — the `error.cause` chain of each captured error.
**Server context the kept integrations attach**
- `RequestData` — attaches nothing under this feature's configuration. See
"Server request data" above for the verified reason.
- `ChildProcess` — a non-zero child-process exit becomes a breadcrumb.
- `Modules` and `Context` — the dependency list, the host name, the
operating system, and the runtime version.
- `ProcessSession` — one release-health session for each process.
- `LocalVariablesAsync` — off. It needs `includeLocalVariables: true`,
which this feature omits.
- `NodeSystemError` — a Node system error (for example, `ENOENT`) gets a
`node_system_error` context field with its error code. The `path` and
`dest` fields are removed by default.
**Server sources this feature removes**
- `Console` — raw `console.*` arguments.
- `ContextLines` — 7 local source lines around each stack frame.
- The outbound breadcrumb of `Http` — outbound request URLs and query
strings.
**Browser events this feature adds**
- A crash in the application error boundary and a crash in the route
error boundary.
**Browser events and context the kept integrations add**
- `GlobalHandlers` — `window.onerror` and `window.onunhandledrejection`.
- `BrowserApiErrors` — a throw inside `setTimeout`, `setInterval`,
`requestAnimationFrame`, and an event listener.
- `CultureContext` — the locale and the timezone.
- `Dedupe`, `LinkedErrors`, and `BrowserSession`.
**Browser sources this feature removes**
- `HttpContext` — the page URL, the referrer, and the user agent.
- `Breadcrumbs` — console output, a click and a keypress target, a
`fetch` and an `XHR` request URL, and history navigation.
**Not captured on either runtime**
- A Zod validation error, which answers 400.
- Each `HttpError` below status 500, such as 401, 403, 404, 409, and 422.
- A performance trace and a profile, because `tracesSampleRate` is 0.
### Operator responsibilities
Two controls belong to the operator. This feature ships neither one.
1. **Set a rate limit and a quota alert.** Set a per-client-key ingestion
rate limit and a quota alert in the Sentry project. The feature sends
no built-in rate limit of its own.
2. **Give a self-hosted sink a reachable host name.** If
`SENTRY_DSN_FRONTEND` (or the legacy `SENTRY_DSN`) points at a
self-hosted Sentry instance, give it an externally reachable ingest
host name, not an internal-only host name. The browser sends its
events from the operator's network, not from the server's network, so
an internal-only host name fails silently for the browser even when it
works for the server.
## Sandbox Startup Trace Spans
Paperclip opens OpenTelemetry spans on the sandbox start path. These spans are
an Observability surface. They are not Paperclip Telemetry events. The
generated telemetry contract does not cover them, so this section is their
canonical contract.
The spans are opt-in. Paperclip exports them only when an OTLP endpoint is
configured. With no endpoint the whole span path is a no-op. Paperclip opens the
spans only for a run that targets a remote sandbox. A local run and an SSH run
stay out of these spans.
Every span attribute uses the closed `paperclip.sandbox.startup.` prefix and
rides a fixed allowlist. A command line, an argument, an environment value, a
file path, program output, or a raw identifier never rides a span. It rides
neither as an attribute nor as an event. The producer bounds each free-form
value:
- A command basename maps to a small known set. Any other value maps to `other`.
- A region maps to a small known set. Any other value maps to `unknown`.
- An image id, a sandbox id, and a lease id ride only as a non-reversible short
hash.
Each numeric attribute is finite. Paperclip omits an attribute when its value is
absent, never a misleading `0`.
### Spans
| Span | Scope | Parent |
| ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- |
| `sandbox.startup` | The one root span for a sandbox bring-up. | none (root) |
| `workspace.resolve` | Workspace resolution step. | `sandbox.startup` |
| `codex-home.seed` | Managed-home seed step. | `sandbox.startup` |
| `skills.reconcile` | Skills reconcile step. | `sandbox.startup` |
| `stage.sync` | Workspace stage-sync step. | `sandbox.startup` |
| `snapshot.git` | Host-side git workspace enumeration inside `stage.sync` (`git status --ignored`, the HEAD diffs, `ls-files`). | `stage.sync` |
| `snapshot.baseline` | Host-side baseline workspace content-hash walk inside `stage.sync`, kept for restore. | `stage.sync` |
| `stage.workspace` | One inbound workspace stage task inside `stage.sync`. It packs and uploads the workspace. | `stage.sync` |
| `stage.asset.<key>` | One inbound asset stage task inside `stage.sync`. It packs and uploads one managed-home asset. The `<key>` segment is the asset key. | `stage.sync` |
| `stage.project.<id>` | One inbound referenced-project stage task inside `stage.sync`. It uploads one referenced project. The `<id>` segment is the project id. | `stage.sync` |
| `pack` | Host-side workspace tarball build inside the `stage.workspace` task. | `stage.workspace` |
| `bridge.paperclip` | Paperclip bridge start step. | `sandbox.startup` |
| `bridge.process-session` | Process-session bridge start step. | `sandbox.startup` |
| `acp.handshake` | ACP session handshake step. | `sandbox.startup` |
| `sandbox.syncBack` | The settlement sync-back that restores the managed home at teardown. | the active run span |
| `restore.workspace` | One outbound workspace restore task at teardown. It reads the sandbox workspace back and merges it into the host workspace. | `sandbox.syncBack` |
| `restore.asset.<key>` | One outbound asset restore task at teardown. It reads one asset back to its host store. The `<key>` segment is the asset key. | `sandbox.syncBack` |
| `sandbox.agentSession.sendInput` | One outbound ACP message to the agent — the socket handler's one `writeTextFile` exec. | the active run span |
| `sandbox.agentSession.pollOutput` | One 100 ms poll tick — `list`, then `read`+`remove` per file found (`1 + 2n` execs). | the active run span |
| `sandbox.callbackBridge.relayRequest` | One Paperclip-API callback request — read the request, write the response, remove it. | the active run span |
| `sandbox.agentProcess` | The persistent streamed agent process the process-session bridge launches; open until the process settles or the bridge tears down, whichever comes first. | the active run span |
| `sandbox.exec` | One host-to-sandbox execution. | the active step or wrapper span |
A step span name is the step name. The `sandbox.exec` span parents to the step
span that runs the execution, so each execution nests under its step. Within
`stage.sync`, the host-side sub-steps `snapshot.git` and `snapshot.baseline` open
as child spans of the step, so the host work at the head of the step is
attributed rather than showing as a gap. Each inbound sync operation also opens
its own task span under `stage.sync`: `stage.workspace`, one `stage.asset.<key>`
per asset, and one `stage.project.<id>` per referenced project. The `pack` span
nests under `stage.workspace`, because the host builds the tarball inside that
task. Two concurrent tasks produce overlapping spans.
The settlement `sandbox.syncBack` span runs at teardown and parents to the run
span. It wraps the managed-home restore. Each outbound restore operation opens
its own task span under `sandbox.syncBack`: `restore.workspace` and one
`restore.asset.<key>` per asset. Two concurrent restore tasks produce overlapping
spans. A run-time
`sandbox.exec` span parents instead to the run-time wrapper span that runs it
(`sandbox.agentSession.sendInput`, `sandbox.agentSession.pollOutput`,
`sandbox.callbackBridge.relayRequest`, or `sandbox.agentProcess`). Each run-time
wrapper span parents to the live run span (`agent.turn` during the turn,
`task.run` otherwise). With no active trace context the exec span opens
unparented.
`sandbox.agentProcess` wraps the persistent streamed agent process. The
process-session bridge launches it during `bridge.process-session`, so it opens
under `task.run` — no turn has started yet. It therefore overlaps the sibling
`agent.turn` rather than nesting under it or dangling off the short-lived bring-up
step. The span ends when the process settles or when the bridge tears down,
whichever comes first. The bridge tears down before the run root span ends, so
the span never outlives `task.run` even when the process lingers past teardown
(the sandbox `execute` has no cancel, so a lingering process cannot be forced to
resolve).
The root span sets the error status when the bring-up fails. Each step span sets
the error status when its step fails. The `sandbox.exec` span sets the error
status when the exit code is non-zero or the execution throws.
### Outcome values
The `paperclip.sandbox.startup.outcome` attribute uses a closed value set:
- `ok` — the step or the execution settled with a success result.
- `skipped` — a warm cache skipped the step; the step ran no work.
- `failed` — the step or the execution threw, or the exit code was non-zero.
### Root span attributes
The `sandbox.startup` root span uses this closed attribute allowlist.
| Attribute | Type | Optional | Meaning |
| ---------------------------------------- | ------- | -------- | ---------------------------------------------------------- |
| `paperclip.sandbox.startup.root.wall_ms` | number | no | The root-span wall time of the whole bring-up. |
| `paperclip.sandbox.startup.root.work_ms` | number | no | The sum of the step wall times. |
| `paperclip.sandbox.startup.root.diff_ms` | number | no | `work_ms − wall_ms`; the overlap the parallel steps saved. |
| `paperclip.sandbox.startup.provider` | string | yes | The normalized provider family. |
| `paperclip.sandbox.startup.cold_start` | boolean | yes | Whether the bring-up is a cold start. |
| `paperclip.sandbox.startup.region` | string | yes | The clamped region label. |
| `paperclip.sandbox.startup.image_id` | string | yes | The hashed image id. |
| `paperclip.sandbox.startup.sandbox_id` | string | yes | The hashed sandbox id. |
| `paperclip.sandbox.startup.lease_id` | string | yes | The hashed lease id. |
### Step span attributes
Each bring-up step span uses this closed attribute allowlist. The step name
rides the span name, so no `step` attribute repeats it.
| Attribute | Type | Optional | Meaning |
| ------------------------------------------------------------ | ------ | -------- | -------------------------------------------------------- |
| `paperclip.sandbox.startup.step.wall_ms` | number | no | The wall time of the step. |
| `paperclip.sandbox.startup.outcome` | string | no | The step outcome (`ok`, `skipped`, or `failed`). |
| `paperclip.sandbox.startup.provider` | string | yes | The normalized provider family. |
| `paperclip.sandbox.startup.batch` | string | yes | A shared tag that marks two parallel steps as one batch. |
| `paperclip.sandbox.startup.handshake.create_runtime.wall_ms` | number | yes | The create-runtime sub-time of the `acp.handshake` step. |
| `paperclip.sandbox.startup.handshake.ensure_session.wall_ms` | number | yes | The ensure-session sub-time of the `acp.handshake` step. |
The round-trip count and the provider durations no longer ride a step span. The
per-execution `sandbox.exec` child spans carry that detail.
### `sandbox.exec` span attributes
The `sandbox.exec` span uses this closed attribute allowlist. Paperclip omits a
numeric attribute when the provider does not report the value.
| Attribute | Type | Optional | Meaning |
| ----------------------------------------------- | ------- | -------- | -------------------------------------------------------------------------- |
| `paperclip.sandbox.startup.provider` | string | no | The normalized provider family. |
| `paperclip.sandbox.startup.exec.command` | string | no | The clamped `argv[0]` command label. |
| `paperclip.sandbox.startup.exec.exit_code` | number | yes | The numeric process exit code. |
| `paperclip.sandbox.startup.exec.wall_ms` | number | no | The host-measured wall time of the execution. |
| `paperclip.sandbox.startup.exec.wait_before_ms` | number | yes | The provider handle-fetch wait before the execution ran. |
| `paperclip.sandbox.startup.exec.sandbox_ms` | number | yes | The in-sandbox run time of the execution. |
| `paperclip.sandbox.startup.exec.network_ms` | number | yes | The transport time the host adds; `wall_ms − wait_before_ms − sandbox_ms`. |
| `paperclip.sandbox.startup.exec.critical_path` | boolean | no | Whether the execution sits on the startup critical path. |
| `paperclip.sandbox.startup.exec.cache_hit` | boolean | yes | Whether the provider served the sandbox handle from its warm cache. |
| `paperclip.sandbox.startup.outcome` | string | no | The execution outcome (`ok` or `failed`). |
The plugin decides the cache hit at the sandbox-handle lookup. The span no
longer infers a cache hit from `wait_before_ms == 0`. Paperclip omits the
`cache_hit` attribute when the provider does not report the value.
To add a span attribute, extend the `SANDBOX_STARTUP_SPAN_ATTRS` allowlist in
the code first. Keep the attribute low-cardinality and free of user content.
### Provider spans
A sandbox provider plugin also opens spans for its own sync steps. These spans
use the `sandbox.daytona.` name prefix. They share the
`paperclip.sandbox.startup.` attribute prefix and obey the same opt-in and
no-user-content rules as the startup spans above.
The plugin worker runs in a separate process from the host. So the host treats
every field of a worker-sent span as untrusted input. The host re-clamps the
span name and every attribute at one boundary, the `span.record` host handler,
before it records the span.
| Span | Scope | Parent |
| ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------- |
| `sandbox.daytona.pack` | The host-local pack step that builds the upload tarball. It makes no sandbox round trip. | the active startup step span |
| `sandbox.daytona.transfer` | The transfer step: an upload to the sandbox (inbound) or a download from the sandbox (outbound). The `paperclip.sandbox.startup.transfer.direction` attribute records the direction. | the active sync task span (`stage.*` inbound, `restore.*` under `sandbox.syncBack` outbound) |
| `sandbox.daytona.ensureDirectory` | The `mkdir -p` step that ensures a directory exists before a write. | the active startup step span |
| `sandbox.daytona.checkSymlinkEscape` | The re-check step that a path resolves inside the workspace root before use. | the active startup step span |
| `sandbox.daytona.promote` | The atomic move of a staged temp onto its target via a pinned dir handle. | the active startup step span |
| `sandbox.daytona.extractTarball` | The one round trip that re-checks the path, runs `tar -xf`, and removes the scratch tarball. | the active startup step span |
| `sandbox.daytona.postUploadCommand` | One caller-supplied post-upload command. | the active startup step span |
| `sandbox.daytona.session.open` | The create of the one persistent session for a lease, on the first in-run command. | the active run span |
| `sandbox.daytona.session.close` | The delete of that persistent session on lease release. | the active run span |
| `sandbox.daytona.other` | Any span name outside the known set. | the active startup step span |
The host clamps the span name to the closed set of leaf names above (`pack`,
`transfer`, `ensureDirectory`, `checkSymlinkEscape`, `promote`, `extractTarball`,
`postUploadCommand`, `session.open`, and `session.close`). The host maps a known
name to `sandbox.daytona.<name>`. The host maps any other value to
`sandbox.daytona.other`, so a span name never carries free-form data. Only the
daytona provider emits these spans today, so the segment is the literal
`daytona`.
The `sandbox.daytona.*` spans use this closed attribute allowlist. The host
drops every other key, so a command, an argument, a path, an id, a standard
output, or a standard error never rides a provider span. The host records only
the attributes that the producer sends for one span.
| Attribute | Type | Optional | Meaning |
| ------------------------------------------------ | ------ | -------- | --------------------------------------------------------------------------------------------------------- |
| `paperclip.sandbox.startup.provider` | string | no | The normalized provider family. |
| `paperclip.sandbox.startup.outcome` | string | yes | The step outcome (`ok`, `skipped`, or `failed`). |
| `paperclip.sandbox.startup.pack.wall_ms` | number | yes | The host-local wall time of the pack step. It rides the `sandbox.daytona.pack` span. |
| `paperclip.sandbox.startup.transfer.wall_ms` | number | yes | The wall time of the transfer step. It rides the `sandbox.daytona.transfer` span. |
| `paperclip.sandbox.startup.transfer.guard.count` | number | yes | The number of serial guard round trips before one transfer. It rides the `sandbox.daytona.transfer` span. |
| `paperclip.sandbox.startup.transfer.direction` | string | yes | The transfer direction (`inbound` or `outbound`). It rides the `sandbox.daytona.transfer` span. |
The `span.record` host handler enforces the allowlist. It re-maps `provider`
through the provider-family normalizer. It keeps `outcome` only when the value
is `ok`, `skipped`, or `failed`. It keeps `transfer.direction` only when the
value is `inbound` or `outbound`. It keeps a numeric attribute only when the
value is a finite number. It drops a status message and keeps only the numeric
status code. The handler never throws, because observability must not change the
sync control flow.
The `span.record` host method needs the `environment.drivers.register`
capability. So only a plugin that registers an environment driver may emit a
provider span. The capability gate rejects a provider span from any other
plugin.
The host parents each provider span to the active sync task span. An inbound
transfer runs inside a `stage.*` task span, so its provider spans parent there.
An outbound transfer runs inside a `restore.*` task span under `sandbox.syncBack`
at teardown, so its provider spans parent there. The host mints a W3C
`traceparent` from the active task span and passes it to the plugin worker on the
per-call invocation channel. The teardown restore runs inside the run-parented
`sandbox.syncBack` span, so the host mints a `traceparent` for an outbound
provider span the same way it does for an inbound one. The worker tags its span with the
`traceparent` and treats the value as opaque. The worker never derives the
parent from it. The host recovers the `traceparent` from its own invocation
record, so a worker can never forge a parent. The host validates the
`traceparent` and rejects a missing or malformed value. With no active host
trace context the worker sends no span, so the whole provider-span path is a
no-op.
## Sandbox Duplex Transport Instrumentation
This section documents one duplex transport with three sinks: an
OpenTelemetry span, a counter in the `tool_runtime_metric_counters` table, and
one run-log event.
Paperclip opens a fixed observability surface for the sandbox duplex transport.
This instrumentation is separate from Paperclip Telemetry events and from the
sandbox startup trace spans above. The generated telemetry contract does not
cover it, so this section is its canonical contract. The code owner is
`packages/adapter-utils/src/duplex-observability.ts`. That module holds each name and
each enum value as a literal constant, so the surface never drifts.
The surface is opt-in. The host injects a recorder that binds the span to the
OTel tracer, the counter to the guarded counter store in
`server/src/services/tool-runtime-metrics.ts`, and the event to the run-events
bridge. The default recorder is a no-op, so the whole surface stays inert until
the host binds a real recorder. Every recorder call sits inside an error swallow,
so a telemetry failure never breaks the request path.
The surface carries no user content. No route, no query, no request body, no
token, and no raw identifier rides a span, a counter, or an event. Each record
carries only the closed dimension keys below and, for the request span, a
latency. The `provider` dimension carries only the allowlisted public value
`daytona`. Any other plugin key maps to `other` before the record reaches a sink,
so a raw plugin key never reaches a span attribute, a counter label, or an event
field.
### Spans
| Span | Scope | Latency |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------ |
| `sandbox.duplex.channel_open` | One duplex channel-open attempt. The `outcome` dimension is `ok` when the channel opened and readiness passed, or `error` when the open or readiness failed. | none |
| `sandbox.duplex.request` | One duplex request the broker forwarded to the host. | The request latency in milliseconds. |
### Event
| Event | Scope |
| -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `sandbox.duplex.transport` | The host emits it at each transport boundary: a ready duplex channel, a fallback to the file bridge, and a terminal channel loss. Its dimensions record the boundary. |
### Counters
| Counter | Scope |
| ----------------------------------- | ----------------------------------------------------------------------------------- |
| `sandbox_duplex_channel_open_total` | One successful duplex channel open. |
| `sandbox_duplex_fallback_total` | One fallback to the file bridge. The `fallback_reason` dimension records the cause. |
| `sandbox_duplex_loss_total` | One terminal duplex channel loss. The `loss_class` dimension records the phase. |
| `sandbox_duplex_session_leak_total` | One leaked provider session at teardown. |
### Dimension keys
Counters carry no dimension labels. The guarded counter store keys each counter
on `(companyId, metric)` with no label column, so the `fallback_reason` and
`loss_class` values fold into the counter metric name instead. The full closed
dimension set below rides only the spans and the `sandbox.duplex.transport`
event, which use only these closed keys. A test asserts the exact set, so a new
key never reaches a sink by accident.
| Key | Type | Optional | Value set |
| --- | --- | --- | --- |
| `provider` | string | no | `daytona`, or `other` for any other plugin key. |
| `transport` | string | no | `duplex`, `http2`, or `file`. `duplex` names the retired bespoke frame protocol; `http2` names the Node HTTP/2 session over the sandbox channel; a fallback record uses `file`. |
| `outcome` | string | yes | `ok` or `error`. |
| `fallback_reason` | string | yes | `gate_off`, `capability_absent`, `route_busy`, `entrypoint_sync_failed`, `broker_construction_failed`, `channel_open_failed`, `ready_invalid`, `ready_nonce_mismatch`, `ready_timeout`, `contaminated`, or `preface_missing`. It rides only a fallback record. `route_busy` marks the process-scoped route ceiling full. `entrypoint_sync_failed` and `broker_construction_failed` mark the named build step. `channel_open_failed` marks a failed channel open. `preface_missing` marks a missing or an invalid HTTP/2 client connection preface inside the bounded readiness buffer: the host found no valid preface after the accepted READY line, aborted the `http2` open, and moved the run to the file bridge (`queue_v1`) one time. |
| `loss_class` | string | yes | `pre_dispatch` or `post_dispatch`, relative to the first request dispatch. It rides only a loss record. |
| `loss_reason` | string | yes | `stdin_eof`, `provider_exit`, `heartbeat_timeout`, `rpc_failure`, `write_error`, `transport_closed`, or `other`. The host maps every loss cause to one of these values, so no raw provider text reaches a sink. `write_error` marks a rejected host-to-sandbox write. `transport_closed` marks a reason-less provider transport close with no exit data. It rides only a loss record. |
To add a name or an enum value, extend the literal constant in
`duplex-observability.ts` first, then update the test that asserts the closed set.
### Known behavior: aggregate retained body bytes
Each HTTP/2 bridge route holds up to 168,820,736 bytes (161 MiB) at its own
peak (see `HTTP2_BRIDGE_MAX_CONCURRENT_STREAMS` in `http2-bridge-server.ts`).
The host process admits up to 128 concurrent routes (see
`DEFAULT_MAX_CONCURRENT_DUPLEX_ROUTES` in `plugin-worker-manager.ts`). Those
two figures alone would let the process retain up to 21,609,054,208 bytes
(about 20.1 GiB) of body data across every route at the same time.
The process does not reach that figure, on two levels.
`HTTP2_BRIDGE_MAX_PROCESS_BODY_BYTES` (`http2-bridge-server.ts`) enforces a
real, live ledger: 1,073,741,824 bytes (1 GiB) across every route, not merely
an accepted paper ceiling. Every HTTP/2 stream creates one `BridgeBodyReservation` owner over
its lifetime, and every source-level full-body buffer that stream retains —
its request-body chunk array, the concatenated request body, the
response-body chunk array, and the concatenated response body — reserves
against that one owner before it allocates. A reservation that would pass the
process total is denied before it copies anything, and the host answers 503
instead of accepting the body. The reservation stays live for the response
body until the HTTP/2 write actually finishes flowing to the peer or the
stream closes, not merely until the write call returns, so a slow or
backpressured peer cannot hold response bytes in memory the ledger no longer
counts.
`HTTP2_BRIDGE_MAX_ROUTE_BODY_BYTES` adds a second, per-route ledger on top of
that process-wide one: each route's own reservations also check a ceiling
scoped to that one route (its own 168,820,736-byte peak from above), so one
busy or malicious route can pass its own ceiling and get denied with a 503,
but it can never spend the whole process-wide total and deny every sibling
route admission. This accounting covers source-level full-body buffers only:
internal Node.js and Undici copies (socket buffers, HTTP/2 frame buffers,
decompression buffers) stay outside it.
The generated gateway process inside the sandbox (`getSandboxCallbackBridgeServerSource`
in `sandbox-callback-bridge.ts`) enforces its own separate ledger, independent
of the two host-side ledgers above: each side bounds only the memory in its
own process. `readBodyBytes` reserves a request body's chunk bytes as they
arrive, then reserves the concatenated buffer's own byte count before
`Buffer.concat` allocates it, against a ceiling of `maxBodyBytes * 8` (4
concurrent bodies, each counted twice for its two live copies). A denied
reservation answers 503 with no forward call. Each request handler releases
its own reservation once the whole request settles: a completed response, a
thrown error, a client abort, or a deadline timeout all reach the same
release call.
Keep every dimension low-cardinality and free of user content.
### Shared skill preparation
`skills.prepare` measures the shared inventory listing and runtime materialization
inside `task.prepare`. It is also contained in the broader
`heartbeat.prepare_before_environment` interval; do not add those two durations.
Preparation failures emit a failed span even when no native session starts.
It carries no skill contents, identifiers, locations, or credentials. It uses the
existing run performance events and operator-configured OpenTelemetry endpoint;
no first-party Telemetry event is added.
Runtime preparation refreshes the company inventory once per listing. Local and
catalog directories remain direct sources, so edits are visible on the next
preparation. Explicit version selections still use their stored snapshots.
Reconstructed skills use `__runtime_cache_v1__/<skill-id>/<fingerprint>/files`
beneath company skill storage, with a sibling manifest of paths, sizes, and SHA-256
content digests. Every warm hit validates the manifest and exact file contents;
it does not fetch upstream, rewrite files, or remove directories. The fingerprint
includes installed source identity, revision, file inventory, and stored Markdown,
and excludes display names, stars, and general update timestamps. Manifests stay
outside the directory delivered to agents.
GitHub and skills.sh imports are cached only when pinned to a full commit SHA.
Remote freshness is explicit: update or reimport selects a new revision, including
supporting-file-only changes. A branch advancing upstream does not change an
installed revision. Legacy mutable refs retain uncached behavior until updated.
URL-only skills use stored Markdown. An unavailable new revision reports missing;
it never silently reuses an older revision. Stored `SKILL.md` remains a fallback,
but missing supporting files prevent publication of a reusable partial cache.
Builds publish read-only files and directories from unique staging directories.
A skill-scoped lock serializes builds and cleanup across processes. Cold builders
recheck that the skill still exists under its original key before reading files
and before atomic publication. Existing valid
revisions stay readable during updates. Invalid entries are quarantined in the
same skill cache root for inspection; rename/removal cleans up that skill's cache.
Read-only listings validate caches without downloading or repairing them. A
publication lock left by an abruptly terminated process is reported for operator
cleanup; remove it only after confirming its recorded PID is no longer running.
Run `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-skill-preparation.ts` for an isolated embedded
PostgreSQL benchmark with 114 mixed skills and at least 400 remote files. It
reports one cold sample and ten warm samples (one in a new process), refresh and
fetch counts, rebuilds, missing entries, and content checks. Upstream responses are
deterministic fixtures; use real deployed run spans for user-facing latency.
Reference in New Issue View Git Blame Copy Permalink