RomanNum3ral
/
paperclip
mirror of https://github.com/paperclipai/paperclip.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

feat: activate OpenTelemetry spans on the sandbox start path (#10536) 

## Thinking Path

> - Paperclip moves agent work through sandboxed execution and
control-plane services.
> - The sandbox start path now has a no-op span seam.
> - This change turns that seam on when OTLP export is configured.
> - It keeps the default path unchanged when export is off.
> - The result is structured startup traces with low-cardinality
attributes and explicit parent links.
> - The benefit is better observability without changing normal
behavior.

## Linked Issues or Issue Description

No public GitHub issue exists for this change.

### Problem
The sandbox start path has a tracer seam, but it stays a no-op unless
the OTLP export path is active.

### Proposed solution
Enable the server tracer on sandbox bring-up, open a root span, parent
each startup boundary to that root, and keep the export path opt-in
behind `OTEL_EXPORTER_OTLP_ENDPOINT`.

### Alternatives considered
- Keep the start path as a no-op. I rejected that path because it leaves
sandbox start opaque when OTLP export is already configured.
- Add broad attributes for commands and paths. I rejected that path
because the span allowlist must stay low-cardinality.

### Roadmap alignment
This follows the current OTel sandbox-start work and keeps the default
path unchanged.

## What Changed

- Add a root sandbox startup span and child spans for each named startup
boundary.
- Keep concurrent bridge spans parented to the root span.
- Inject the server tracer through the adapter deps without
OpenTelemetry imports in the engine.
- Attach host-received provider duration attributes only when the values
are finite.
- Keep span attributes inside the allowlist and keep command, path, id,
and error text out of span data.

## Verification

- The pushed branch already passed `pnpm --filter
@paperclipai/adapter-utils exec tsc --noEmit`.
- The pushed branch already passed `pnpm exec vitest run
packages/adapter-utils/src/acpx-engine/`.
- The pushed branch already passed `pnpm --filter @paperclipai/server
exec vitest run src/__tests__/environment-execution-target.test.ts
src/__tests__/instrumentation.test.ts`.
- The pushed branch already passed `pnpm --filter @paperclipai/server
exec tsc --noEmit`.

## Risks

- OTel export changes trace volume when the endpoint is set.
- The allowlist limits trace detail, so new fields need care.
- The change stays no-op when OTLP export is off.

## Model Used

- OpenAI GPT-5, tool use enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used with capability details
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue in-PR
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Nicky Leach 2026-07-30 21:20:58 -07:00 committed by GitHub
parent 5cffd5c72e
commit 9f7565f4ce
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
9 changed files with 838 additions and 14 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

295
packages/adapter-utils/src/acpx-engine/execute.test.ts
View File

@ -3,7 +3,7 @@ import os from "node:os";
import path from "node:path";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import type { AcpRuntimeOptions } from "acpx/runtime";
import type { AdapterRuntimeMcpAccess } from "@paperclipai/adapter-utils";
import type { AdapterExecutionContext, AdapterRuntimeMcpAccess } from "@paperclipai/adapter-utils";
import {
DEFAULT_REMOTE_SANDBOX_ADAPTER_TIMEOUT_SEC,
prepareAdapterExecutionTargetRuntime,
@ -167,6 +167,7 @@ async function runExecutor(
executionTarget?: Record<string, unknown>;
runtimeMcp?: AdapterRuntimeMcpAccess;
prepareRemoteManagedHome?: AcpxEngineExecutorOptions["prepareRemoteManagedHome"];
startupTraceContext?: AdapterExecutionContext["startupTraceContext"];
} = {},
) {
const runtimeOptions: Record<string, unknown>[] = [];
@ -201,6 +202,7 @@ async function runExecutor(
authToken: options.authToken,
executionTarget: options.executionTarget,
runtimeMcp: options.runtimeMcp,
startupTraceContext: options.startupTraceContext,
onLog: async (stream: "stdout" | "stderr", text: string) => {
logs.push({ stream, text });
},
@ -216,6 +218,77 @@ async function runExecutor(
return { logs, meta, events, runtimeOptions, configOptions, sessionInputs, result };
}
// A recording span, used only in tests. It captures the span name, the parent
// span (resolved from the explicit parent-context token), the attribute map,
// the terminal status, and whether the span ended. The engine treats it purely
// through the structural `StartupSpan` contract.
interface RecordingSpan {
name: string;
attributes: Record<string, string | number | boolean>;
parent: RecordingSpan | null;
status: { code: number } | null;
ended: boolean;
setAttribute(key: string, value: string | number | boolean): void;
setStatus(status: { code: number; message?: string }): void;
end(): void;
}
// Build an in-memory startup trace context that records every span. It models
// the real OTel parenting contract: `startSpan(name, options, context)` reads
// the parent from the explicit `context` token that `contextWithSpan` produced,
// so a test asserts the exact parent of each child without an OTel package or
// ambient async-context propagation.
function createRecordingStartupTrace() {
const spans: RecordingSpan[] = [];
const traceContext = {
tracer: {
startSpan(
name: string,
options?: { attributes?: Record<string, string | number | boolean> },
context?: unknown,
) {
const parent =
context && typeof context === "object" && "span" in context
? ((context as { span: RecordingSpan }).span ?? null)
: null;
const span: RecordingSpan = {
name,
attributes: { ...(options?.attributes ?? {}) },
parent,
status: null,
ended: false,
setAttribute(key: string, value: string | number | boolean) {
span.attributes[key] = value;
},
setStatus(status: { code: number }) {
span.status = { code: status.code };
},
end() {
span.ended = true;
},
};
spans.push(span);
return span;
},
},
contextWithSpan(span: unknown) {
return { span };
},
} satisfies AdapterExecutionContext["startupTraceContext"];
return { traceContext, spans };
}
// The closed span-attribute allowlist for a sandbox-start span (Phase 2 + 3).
// A test asserts every recorded attribute key is in this set, so a command,
// path, id, or error-text key can never ride a span.
const ALLOWED_STARTUP_SPAN_ATTRIBUTE_KEYS = new Set([
"step",
"provider",
"roundTrips",
"providerExecMs",
"providerGetMs",
]);
describe("shared ACPX engine runtime behavior", () => {
it("sets Codex model, effort, and fast mode through CODEX_CONFIG without session config calls", async () => {
const { configOptions, meta } = await runExecutor({
@ -2929,6 +3002,226 @@ describe("ACPX engine remote session-lifecycle re-staging (PR 3: stage once / re
});
});
describe("ACPX engine sandbox-start spans (opt-in root + child parenting)", () => {
beforeEach(() => {
vi.clearAllMocks();
});
async function remoteSandboxTarget(root: string) {
const remoteCwd = path.join(root, "remote-workspace");
await fs.mkdir(remoteCwd, { recursive: true });
return {
kind: "remote",
transport: "sandbox",
// A plugin-backed key (not a built-in family). It must never ride a span
// as a raw attribute.
providerKey: "fake-plugin",
remoteCwd,
runner: createLocalSandboxRunner(),
};
}
it("emits one root span with a child span per bring-up boundary, each parented to the root", async () => {
const root = await makeTempRoot();
const stateDir = path.join(root, "state");
const localCwd = path.join(root, "worktree");
const codexHome = path.join(root, "codex-home");
await fs.mkdir(localCwd, { recursive: true });
await fs.mkdir(codexHome, { recursive: true });
const executionTarget = await remoteSandboxTarget(root);
const { traceContext, spans } = createRecordingStartupTrace();
await runExecutor(
{
agent: "codex",
agentCommand: "node ./fake-acp.js",
stateDir,
cwd: localCwd,
env: { CODEX_HOME: codexHome },
},
{ authToken: "real-run-jwt", executionTarget, startupTraceContext: traceContext },
);
// Exactly one root span, and it is the bring-up root.
const roots = spans.filter((span) => span.parent === null);
expect(roots).toHaveLength(1);
const rootSpan = roots[0]!;
expect(rootSpan.name).toBe("sandbox.startup");
expect(rootSpan.ended).toBe(true);
// A codex bring-up over the remote sandbox lane crosses all 7 boundaries.
const childNames = spans.filter((span) => span !== rootSpan).map((span) => span.name).sort();
expect(childNames).toEqual(
[
"acp.handshake",
"bridge.paperclip",
"bridge.process-session",
"codex-home.seed",
"skills.reconcile",
"stage.sync",
"workspace.resolve",
],
);
// Every child parents to the one root and ends.
for (const span of spans) {
if (span === rootSpan) continue;
expect(span.parent, `span "${span.name}" must parent to the root`).toBe(rootSpan);
expect(span.ended, `span "${span.name}" must end`).toBe(true);
}
});
it("parents both concurrent bridge spans to the root (neither orphans)", async () => {
const root = await makeTempRoot();
const stateDir = path.join(root, "state");
const localCwd = path.join(root, "worktree");
await fs.mkdir(localCwd, { recursive: true });
const executionTarget = await remoteSandboxTarget(root);
const { traceContext, spans } = createRecordingStartupTrace();
await runExecutor(
{ agent: "custom", agentCommand: "node ./fake-acp.js", stateDir, cwd: localCwd },
{ authToken: "real-run-jwt", executionTarget, startupTraceContext: traceContext },
);
const rootSpan = spans.find((span) => span.name === "sandbox.startup" && span.parent === null);
expect(rootSpan).toBeTruthy();
const paperclip = spans.find((span) => span.name === "bridge.paperclip");
const processSession = spans.find((span) => span.name === "bridge.process-session");
expect(paperclip?.parent).toBe(rootSpan);
expect(processSession?.parent).toBe(rootSpan);
});
it("keeps every span attribute inside the closed allowlist (no command/path/id keys)", async () => {
const root = await makeTempRoot();
const stateDir = path.join(root, "state");
const localCwd = path.join(root, "worktree");
await fs.mkdir(localCwd, { recursive: true });
const executionTarget = await remoteSandboxTarget(root);
const { traceContext, spans } = createRecordingStartupTrace();
await runExecutor(
{ agent: "custom", agentCommand: "node ./fake-acp.js", stateDir, cwd: localCwd },
{ authToken: "real-run-jwt", executionTarget, startupTraceContext: traceContext },
);
expect(spans.length).toBeGreaterThan(0);
for (const span of spans) {
for (const [key, value] of Object.entries(span.attributes)) {
expect(
ALLOWED_STARTUP_SPAN_ATTRIBUTE_KEYS.has(key),
`span "${span.name}" set a non-allowlisted attribute "${key}"`,
).toBe(true);
// No non-finite numeric attribute (no NaN, no Infinity).
if (typeof value === "number") {
expect(Number.isFinite(value), `attribute "${key}" must be finite`).toBe(true);
}
}
}
});
it("closes the root span with error status when the bring-up handshake fails", async () => {
const root = await makeTempRoot();
const stateDir = path.join(root, "state");
const localCwd = path.join(root, "worktree");
await fs.mkdir(localCwd, { recursive: true });
const executionTarget = await remoteSandboxTarget(root);
const { traceContext, spans } = createRecordingStartupTrace();
const execute = createAcpxEngineExecutor({
createRuntime: () =>
({
ensureSession: async () => {
throw new Error("handshake boom");
},
startTurn: () => ({
events: (async function* () {})(),
result: Promise.resolve({ status: "failed" }),
cancel: async () => {},
}),
setConfigOption: async () => {},
close: async () => {},
}) as never,
});
const result = await execute({
runId: "run-handshake-fail",
agent: { id: "agent-1", companyId: "company-1" },
runtime: {},
config: { agent: "custom", agentCommand: "node ./fake-acp.js", stateDir, cwd: localCwd },
context: {},
authToken: "real-run-jwt",
executionTarget,
startupTraceContext: traceContext,
onLog: async () => {},
onMeta: async () => {},
onEvent: async () => {},
} as never);
expect(result.exitCode).toBe(1);
const rootSpan = spans.find((span) => span.name === "sandbox.startup" && span.parent === null);
expect(rootSpan).toBeTruthy();
expect(rootSpan!.ended).toBe(true);
// `2` is `SpanStatusCode.ERROR`.
expect(rootSpan!.status?.code).toBe(2);
});
it("opens no span and does not throw when no trace context is injected", async () => {
const root = await makeTempRoot();
const stateDir = path.join(root, "state");
const localCwd = path.join(root, "worktree");
await fs.mkdir(localCwd, { recursive: true });
const executionTarget = await remoteSandboxTarget(root);
// runExecutor asserts exitCode 0 internally; the run must complete with no
// injected trace context (the default no-op path).
const { result } = await runExecutor(
{ agent: "custom", agentCommand: "node ./fake-acp.js", stateDir, cwd: localCwd },
{ authToken: "real-run-jwt", executionTarget },
);
expect(result.exitCode).toBe(0);
});
it("opens no span for a local run, even when a trace context is injected", async () => {
const root = await makeTempRoot();
const stateDir = path.join(root, "state");
const localCwd = path.join(root, "worktree");
await fs.mkdir(localCwd, { recursive: true });
const { traceContext, spans } = createRecordingStartupTrace();
// A local run has no execution target. The `sandbox.startup` span names a
// sandbox bring-up, so a local run must stay out of sandbox telemetry.
const { result } = await runExecutor(
{ agent: "custom", agentCommand: "node ./fake-acp.js", stateDir, cwd: localCwd },
{ authToken: "real-run-jwt", startupTraceContext: traceContext },
);
expect(result.exitCode).toBe(0);
expect(spans).toHaveLength(0);
});
it("opens no span for an SSH run, even when a trace context is injected", async () => {
const root = await makeTempRoot();
const stateDir = path.join(root, "state");
const localCwd = path.join(root, "worktree");
const remoteCwd = path.join(root, "remote-workspace");
await fs.mkdir(localCwd, { recursive: true });
const { traceContext, spans } = createRecordingStartupTrace();
// An SSH run is remote but is not a sandbox, so it also stays out of sandbox
// telemetry.
const { result } = await runExecutor(
{ agent: "custom", agentCommand: "node ./fake-acp.js", stateDir, cwd: localCwd },
{
authToken: "real-run-jwt",
executionTarget: { kind: "remote", transport: "ssh", remoteCwd },
startupTraceContext: traceContext,
},
);
expect(result.exitCode).toBe(0);
expect(spans).toHaveLength(0);
});
});
describe("ACPX engine per-step startup timing (run.startup.step events)", () => {
beforeEach(() => {
vi.clearAllMocks();

126
packages/adapter-utils/src/acpx-engine/execute.ts
View File

@ -83,7 +83,15 @@ import {
DEFAULT_ACP_ENGINE_TIMEOUT_SEC,
DEFAULT_ACP_ENGINE_WARM_HANDLE_IDLE_MS,
} from "./constants.js";
import { measureStartupStep, type StartupStepMeasureOptions } from "./startup-timing.js";
import {
measureStartupStep,
NOOP_STARTUP_SPAN,
NOOP_STARTUP_TRACE_CONTEXT,
type StartupSpan,
type StartupSpanContext,
type StartupStepMeasureOptions,
type StartupTraceContext,
} from "./startup-timing.js";
import type { CommandManagedRuntimeRunner } from "../command-managed-runtime.js";
const defaultModuleDir = path.dirname(fileURLToPath(import.meta.url));
@ -1335,6 +1343,10 @@ async function buildRuntime(input: {
ctx: AdapterExecutionContext;
engine: AcpxEngineSettings;
deps: AcpxEngineExecutorOptions;
// The injected tracer plus the root-span parent-context token. Merged into
// every startup-step option set, so each boundary span parents to the one
// root span (`sandbox.startup`) that the executor opens.
spanParent: Pick<StartupStepMeasureOptions, "tracer" | "parentContext">;
}): Promise<AcpxPreparedRuntime> {
const { runId, agent, config, context, authToken } = input.ctx;
// Injectable monotonic clock for per-step startup timing. Hoisted above the
@ -1410,17 +1422,23 @@ async function buildRuntime(input: {
// and emits the per-step delta. Empty when there is no runner (local runs,
// the runner-less ACP→CLI fallback, or an SSH runner that does not
// instrument the seam), so those steps simply omit the fields.
const stepMetrics = buildStartupStepMetrics(
executionTarget?.kind === "remote" && executionTarget.transport === "sandbox"
? executionTarget.runner
: undefined,
);
// Merge the injected tracer + root parent-context into every step option set,
// so each boundary span parents to the root span. With no injected trace
// context both fields are no-ops and the span path stays inert.
const stepMetrics: StartupStepMeasureOptions = {
...buildStartupStepMetrics(
executionTarget?.kind === "remote" && executionTarget.transport === "sandbox"
? executionTarget.runner
: undefined,
),
...input.spanParent,
};
// The two bridge-start steps intentionally overlap, so their runner counters
// would double-count each other if we sampled them here. Keep the shared
// counter attribution on the sequential startup phases only; the concurrent
// bridge steps still emit duration telemetry, just not misleading per-step
// round-trip/provider deltas.
const concurrentBridgeStepMetrics: StartupStepMeasureOptions = {};
// bridge steps still emit duration telemetry (and a span), just not
// misleading per-step round-trip/provider deltas.
const concurrentBridgeStepMetrics: StartupStepMeasureOptions = { ...input.spanParent };
const shapedWorkspaceEnv = shapePaperclipWorkspaceEnvForExecution({
workspaceCwd: effectiveWorkspaceCwd,
workspaceWorktreePath,
@ -2787,6 +2805,54 @@ function warmHandleMatches(
return entry !== undefined && entry.runtime === runtime && entry.handle === handle;
}
/** The stable name of the one root span for a sandbox bring-up. It is a fixed
* low-cardinality constant, never derived from run/user data. */
const STARTUP_ROOT_SPAN_NAME = "sandbox.startup";
/**
* Open the one root span for a sandbox bring-up and return its parent-context
* token plus a guarded `end`. The span parents every startup boundary span:
* the engine forwards `parentContext` to each `measureStartupStep` call. The
* `end` closure runs at most once (bring-up complete OR a bring-up failure) and
* swallows every tracer error, so observability never changes startup control
* flow. With no injected trace context, the tracer is a no-op and the span is
* a no-op.
*/
function openStartupRootSpan(tracing: StartupTraceContext): {
parentContext: StartupSpanContext;
end: (failed: boolean) => void;
} {
let span: StartupSpan;
try {
span = tracing.tracer.startSpan(STARTUP_ROOT_SPAN_NAME);
} catch {
span = NOOP_STARTUP_SPAN;
}
let parentContext: StartupSpanContext;
try {
parentContext = tracing.contextWithSpan(span);
} catch {
parentContext = undefined;
}
let ended = false;
return {
parentContext,
end: (failed: boolean) => {
if (ended) return;
ended = true;
try {
// `2` is `SpanStatusCode.ERROR`. `adapter-utils` stays OTel-free, so it
// uses the numeric value that a real injected span reads as the error
// status.
if (failed) span.setStatus({ code: 2 });
span.end();
} catch {
// Observability must not change startup control flow.
}
},
};
}
export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) {
const createRuntime = deps.createRuntime ?? createAcpRuntime;
const now = deps.now ?? (() => Date.now());
@ -2817,7 +2883,39 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) {
now,
idleMs: warmIdleMs,
});
const prepared = await buildRuntime({ ctx, engine, deps });
// The `sandbox.startup` span names a sandbox bring-up. It must not cover a
// local or SSH run: those runs have no sandbox, so they stay out of sandbox
// telemetry. Open the real root span only when the target is a remote
// sandbox; every other target forces the no-op trace context, so the whole
// startup span path stays inert regardless of the injected context.
const startupExecutionTarget = readAdapterExecutionTarget({
executionTarget: ctx.executionTarget,
legacyRemoteExecution: ctx.executionTransport?.remoteExecution,
});
const targetsRemoteSandbox =
startupExecutionTarget?.kind === "remote" && startupExecutionTarget.transport === "sandbox";
// Open the one root span for this bring-up. It spans `buildRuntime` through
// `acp.handshake`, so every startup boundary span parents to it. `spanParent`
// carries the injected tracer + the root parent-context token into each
// `measureStartupStep` call. With no injected trace context the whole path
// is a no-op. `endRootSpan` runs exactly once — at bring-up completion or on
// a bring-up failure.
const tracing =
targetsRemoteSandbox && ctx.startupTraceContext
? ctx.startupTraceContext
: NOOP_STARTUP_TRACE_CONTEXT;
const rootSpan = openStartupRootSpan(tracing);
const spanParent: Pick<StartupStepMeasureOptions, "tracer" | "parentContext"> = {
tracer: tracing.tracer,
parentContext: rootSpan.parentContext,
};
let prepared: AcpxPreparedRuntime;
try {
prepared = await buildRuntime({ ctx, engine, deps, spanParent });
} catch (err) {
rootSpan.end(true);
throw err;
}
// State the effective wall-clock timeout and its source up front so a
// later timeout is diagnosable from the run log alone. Goes to stderr:
// the acpx stdout log stream carries JSON acpx.* event payloads and must
@ -2945,6 +3043,8 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) {
}
}
} catch (err) {
// Bring-up failed at the handshake — close the root span with error status.
rootSpan.end(true);
const { classified, message } = await emitAcpxFailure({
ctx,
prepared,
@ -2968,6 +3068,8 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) {
}
if (!handle) {
// Bring-up produced no session handle — close the root span with error status.
rootSpan.end(true);
await discardStagedRuntime({ handles: stagedRuntimes, prepared });
await cleanupRemoteBridges(prepared);
return {
@ -2982,6 +3084,10 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) {
summary: "ACPX did not return a runtime session handle.",
};
}
// Bring-up is complete: the session handle is established. Close the root
// span here, so it covers `buildRuntime` through `acp.handshake` and no
// further. The agent turn runs after and is out of the startup root's scope.
rootSpan.end(false);
const sessionHandle = handle;
try {
await applySessionConfigOptions({

54
packages/adapter-utils/src/acpx-engine/startup-timing.ts
View File

@ -57,25 +57,58 @@ export interface StartupSpan {
end(): void;
}
/**
* An opaque parent-context token. The server builds it from the OTel
* `@opentelemetry/api` `context` / `trace` helpers. `adapter-utils` never reads
* it; it only forwards it to `startSpan`, so this package stays OTel-free. A
* child span opened with this token parents to the span the token carries.
*/
export type StartupSpanContext = unknown;
/**
* A minimal, OTel-free tracer contract. The server injects a real
* `@opentelemetry/api` tracer, which satisfies this shape structurally. The
* `startSpan` signature is a subset of the OTel one, so a real tracer is
* assignable here.
* assignable here. The optional third argument is the explicit parent context:
* a real OTel `startSpan(name, options, context)` parents the new span to the
* span that `context` carries. `adapter-utils` passes it through as an opaque
* token, so parenting never depends on ambient async-context propagation.
*/
export interface StartupTracer {
startSpan(
name: string,
options?: { attributes?: Record<string, string | number | boolean> },
context?: StartupSpanContext,
): StartupSpan;
}
const NOOP_SPAN: StartupSpan = {
/**
* The injected tracer plus the one context helper the engine needs to build a
* parent-context token from the root span. The server binds these to
* `@opentelemetry/api` (`trace.getTracer`, `trace.setSpan` over
* `context.active()`). The default is a no-op, so the whole span path stays a
* no-op until the server injects a real implementation.
*/
export interface StartupTraceContext {
readonly tracer: StartupTracer;
/**
* Return a parent-context token whose active span is `span`. A child span
* opened with this token parents to `span`. The token is opaque to
* `adapter-utils`.
*/
contextWithSpan(span: StartupSpan): StartupSpanContext;
}
/** A shared no-op span. It implements the structural span contract and does
* nothing, so a caller with no injected tracer changes no behavior. */
export const NOOP_STARTUP_SPAN: StartupSpan = {
setAttribute() {},
setStatus() {},
end() {},
};
const NOOP_SPAN = NOOP_STARTUP_SPAN;
/**
* The default tracer. It opens no real span, so `measureStartupStep` behaves
* exactly as before when the caller injects no tracer.
@ -84,6 +117,16 @@ const NOOP_TRACER: StartupTracer = {
startSpan: () => NOOP_SPAN,
};
/**
* The default trace context. Its tracer is a no-op and it produces no parent
* token, so the engine emits no spans until the server injects a real
* implementation.
*/
export const NOOP_STARTUP_TRACE_CONTEXT: StartupTraceContext = {
tracer: NOOP_TRACER,
contextWithSpan: () => undefined,
};
/**
* Set a numeric span attribute only when the value is a finite number. A reader
* that returns `undefined` (the counter is unavailable) yields no attribute,
@ -139,6 +182,10 @@ function finiteDelta(
* span path changes no runtime behavior until the server injects a real
* tracer. The span carries only the closed attribute allowlist (`step`, the
* normalized `provider`, and the finite counter deltas).
* - `parentContext` — an opaque parent-context token from the root span. When
* set, the step's span parents to that root. `measureStartupStep` forwards it
* to `startSpan` and never inspects it, so parenting stays explicit and does
* not depend on ambient async-context propagation.
* - `provider` — the raw provider key for the step. `measureStartupStep`
* normalizes it through `normalizeProviderFamily` before it sets the
* low-cardinality `provider` span attribute. It never sets the raw key.
@ -149,6 +196,7 @@ export interface StartupStepMeasureOptions {
providerGetMs?: () => number;
extra?: () => Record<string, number>;
tracer?: StartupTracer;
parentContext?: StartupSpanContext;
provider?: string;
}
@ -205,7 +253,7 @@ export async function measureStartupStep<T>(
}
let span: StartupSpan;
try {
span = tracer.startSpan(step, { attributes: startAttributes });
span = tracer.startSpan(step, { attributes: startAttributes }, options.parentContext);
} catch {
// A throwing tracer must not change startup control flow.
span = NOOP_SPAN;

8
packages/adapter-utils/src/types.ts
View File

@ -175,6 +175,14 @@ export interface AdapterExecutionContext {
onRuntimeProgress?: RuntimeStatusSink;
onSpawn?: (meta: { pid: number; processGroupId: number | null; startedAt: string }) => Promise<void>;
authToken?: string;
/**
* The injected OpenTelemetry startup trace context (tracer + root
* parent-context helper). The server passes the real, endpoint-gated
* implementation; when absent, the ACPX engine uses a no-op, so the whole
* span path stays inert. The type is an inline import so this module keeps no
* top-level dependency on the timing helper.
*/
startupTraceContext?: import("./acpx-engine/startup-timing.js").StartupTraceContext;
}
export interface AdapterModel {

62
packages/shared/src/telemetry/README.md
View File

@ -55,6 +55,68 @@ If a dimension is privacy-protected before emission, emit only the protected
value and its matching public marker as defined by the typed helper or generated
contract. Do not emit private source material in telemetry dimensions.
## Sandbox Startup Trace Spans
Paperclip opens OpenTelemetry spans on the sandbox start path. These spans are a
separate telemetry surface from the first-party events above. The generated
telemetry contract does not cover them, so this section is their canonical
contract.
The spans are opt-in. Paperclip exports them only when an OTLP endpoint is
configured. With no endpoint the whole span path is a no-op. Paperclip opens the
spans only for a run that targets a remote sandbox. A local run and an SSH run
stay out of these spans.
Span attributes use a closed allowlist. A command, an argument, an environment
value, a file path, an identifier, or program output never rides a span. It rides
neither as an attribute nor as an event. Each numeric attribute is finite.
Paperclip omits an attribute when its value is absent.
### Spans
| Span | Scope | Parent |
| --- | --- | --- |
| `sandbox.startup` | The one root span for a sandbox bring-up. | none (root) |
| `workspace.resolve` | Workspace resolution step. | `sandbox.startup` |
| `codex-home.seed` | Managed-home seed step. | `sandbox.startup` |
| `skills.reconcile` | Skills reconcile step. | `sandbox.startup` |
| `stage.sync` | Workspace stage-sync step. | `sandbox.startup` |
| `bridge.paperclip` | Paperclip bridge start step. | `sandbox.startup` |
| `bridge.process-session` | Process-session bridge start step. | `sandbox.startup` |
| `acp.handshake` | ACP session handshake step. | `sandbox.startup` |
| `provider.execute` | One host-to-sandbox provider exec call. | none |
The root span sets the error status when the bring-up fails. Each step span sets
the error status when its step fails.
### Startup span attributes
The bring-up step spans use this closed attribute allowlist.
| Attribute | Type | Optional | Meaning |
| --- | --- | --- | --- |
| `step` | string | no | The bring-up step name. |
| `provider` | string | yes | The normalized provider family. |
| `roundTrips` | number | yes | Host-to-sandbox round trips for the step. |
| `providerExecMs` | number | yes | Provider-reported exec time for the step, in milliseconds. |
| `providerGetMs` | number | yes | Provider-reported fetch time for the step, in milliseconds. |
### Provider exec span attributes
The `provider.execute` span uses this closed attribute allowlist. Paperclip omits
a duration attribute when the provider does not report the value.
| Attribute | Type | Optional | Meaning |
| --- | --- | --- | --- |
| `provider` | string | no | The normalized provider family. |
| `exit` | string | no | `ok` when the exit code is 0, else `error`. |
| `provider.exec.duration_ms` | number | yes | Provider-reported exec wall time, in milliseconds. |
| `provider.get.duration_ms` | number | yes | Provider-reported fetch wall time, in milliseconds. |
To add a startup span attribute or a provider span attribute, extend the
allowlist in the code first. Keep the attribute low-cardinality and free of user
content.
## Dimension Values
Telemetry dimension values must be primitives. Use only the value types allowed

176
server/src/__tests__/environment-execution-target.test.ts
View File

@ -420,4 +420,180 @@ describe("resolveEnvironmentExecutionTarget", () => {
expect(runner!.providerExecMs()).toBe(0);
expect(runner!.providerGetMs()).toBe(0);
});
// A recording tracer that captures each provider-exec span's name, attribute
// map, and end. It satisfies the structural tracer the seam calls.
function createRecordingExecTracer() {
const spans: Array<{ name: string; attributes: Record<string, unknown>; ended: boolean }> = [];
const tracer = {
startSpan(name: string) {
const span = {
name,
attributes: {} as Record<string, unknown>,
ended: false,
setAttribute(key: string, value: unknown) {
span.attributes[key] = value;
},
end() {
span.ended = true;
},
};
spans.push(span);
return span;
},
};
return { tracer, spans };
}
// The closed span-attribute allowlist for a provider-exec span (Phase 4).
const ALLOWED_EXEC_SPAN_ATTRIBUTE_KEYS = new Set([
"provider",
"exit",
"provider.exec.duration_ms",
"provider.get.duration_ms",
]);
async function runnerFor(input: {
provider: string;
execResult: Record<string, unknown>;
tracer: unknown;
}) {
mockResolveEnvironmentDriverConfigForRuntime.mockResolvedValue({
driver: "sandbox",
config: { provider: input.provider, reuseLease: false, timeoutMs: 30_000 },
});
const environmentRuntime = {
execute: vi.fn().mockResolvedValue(input.execResult),
supportsSync: vi.fn().mockReturnValue(false),
};
const target = await resolveEnvironmentExecutionTarget({
db: {} as never,
companyId: "company-1",
adapterType: "codex_local",
environment: { id: "env-1", driver: "sandbox", config: { provider: input.provider } },
leaseId: "lease-1",
leaseMetadata: { remoteCwd: "/workspace" },
lease: { id: "lease-1" } as never,
environmentRuntime: environmentRuntime as never,
tracer: input.tracer as never,
});
return (target as { runner?: {
execute(input: { command: string; args?: string[] }): Promise<unknown>;
} }).runner!;
}
it("sets the provider duration attributes from finite Daytona-shaped metadata", async () => {
const { tracer, spans } = createRecordingExecTracer();
const runner = await runnerFor({
provider: "daytona",
execResult: {
exitCode: 0,
signal: null,
timedOut: false,
stdout: "ok",
stderr: "",
metadata: { durationMs: 600, getDurationMs: 15 },
},
tracer,
});
await runner.execute({ command: "echo", args: ["a"] });
expect(spans).toHaveLength(1);
const span = spans[0]!;
expect(span.name).toBe("provider.execute");
expect(span.ended).toBe(true);
expect(span.attributes["provider.exec.duration_ms"]).toBe(600);
expect(span.attributes["provider.get.duration_ms"]).toBe(15);
expect(span.attributes.provider).toBe("daytona");
expect(span.attributes.exit).toBe("ok");
});
it("omits each duration attribute when a provider returns no timing (does not throw, keeps provider)", async () => {
const { tracer, spans } = createRecordingExecTracer();
const runner = await runnerFor({
provider: "kubernetes",
execResult: { exitCode: 0, signal: null, timedOut: false, stdout: "", stderr: "" },
tracer,
});
await expect(runner.execute({ command: "echo" })).resolves.toBeTruthy();
expect(spans).toHaveLength(1);
const span = spans[0]!;
expect("provider.exec.duration_ms" in span.attributes).toBe(false);
expect("provider.get.duration_ms" in span.attributes).toBe(false);
// The provider attribute is always present so a trace shows which provider ran.
expect(span.attributes.provider).toBe("kubernetes");
});
it("never emits a `0` duration attribute for a Daytona timeout that omits durationMs", async () => {
const { tracer, spans } = createRecordingExecTracer();
const runner = await runnerFor({
provider: "daytona",
execResult: {
exitCode: 124,
signal: null,
timedOut: true,
stdout: "",
stderr: "",
// The Daytona timeout branch may leave durationMs undefined.
metadata: { getDurationMs: 20 },
},
tracer,
});
await runner.execute({ command: "sleep", args: ["999"] });
const span = spans[0]!;
expect("provider.exec.duration_ms" in span.attributes).toBe(false);
expect(span.attributes["provider.get.duration_ms"]).toBe(20);
expect(span.attributes.exit).toBe("error");
});
it("never sets a command, arg, or non-allowlisted key as an indexed span attribute", async () => {
const { tracer, spans } = createRecordingExecTracer();
const runner = await runnerFor({
provider: "daytona",
execResult: {
exitCode: 0,
signal: null,
timedOut: false,
stdout: "",
stderr: "",
metadata: { durationMs: 5, getDurationMs: 1 },
},
tracer,
});
await runner.execute({ command: "bash -lc 'rm -rf /secret/path'", args: ["--token", "s3cr3t"] });
const span = spans[0]!;
for (const key of Object.keys(span.attributes)) {
expect(ALLOWED_EXEC_SPAN_ATTRIBUTE_KEYS.has(key), `non-allowlisted key "${key}"`).toBe(true);
}
const values = Object.values(span.attributes).map(String);
expect(values.some((value) => value.includes("rm -rf"))).toBe(false);
expect(values.some((value) => value.includes("s3cr3t"))).toBe(false);
});
it("normalizes a plugin-backed provider key to `plugin` and keeps a built-in family as-is", async () => {
const plugin = createRecordingExecTracer();
const pluginRunner = await runnerFor({
provider: "acme-custom-sandbox",
execResult: { exitCode: 0, signal: null, timedOut: false, stdout: "", stderr: "" },
tracer: plugin.tracer,
});
await pluginRunner.execute({ command: "echo" });
expect(plugin.spans[0]!.attributes.provider).toBe("plugin");
const builtIn = createRecordingExecTracer();
const builtInRunner = await runnerFor({
provider: "e2b",
execResult: { exitCode: 0, signal: null, timedOut: false, stdout: "", stderr: "" },
tracer: builtIn.tracer,
});
await builtInRunner.execute({ command: "echo" });
expect(builtIn.spans[0]!.attributes.provider).toBe("e2b");
});
});

69
server/src/instrumentation.ts
View File

@ -97,6 +97,75 @@ export function getStartupTracer(name = "paperclip.startup"): StartupTracerHandl
}
}
/**
* The injected tracer plus the one context helper the ACPX engine needs. The
* engine opens a root span, then builds a parent-context token from it through
* `contextWithSpan`, and forwards that token to each child span. The token
* comes from `trace.setSpan(context.active(), span)`, so the engine never
* imports `@opentelemetry/api`.
*/
export interface StartupTraceContextHandle {
readonly tracer: StartupTracerHandle;
contextWithSpan(span: unknown): unknown;
}
/**
* The no-op trace context returned when `@opentelemetry/api` is absent or a
* lookup fails. Its tracer is a no-op and it produces no parent token, so
* startup tracing stays a no-op without an installed OTel package.
*/
const NOOP_TRACE_CONTEXT: StartupTraceContextHandle = {
tracer: NOOP_TRACER,
contextWithSpan: () => undefined,
};
let traceContextApiLoadFailed = false;
/**
* Return a startup trace context (tracer + root parent-context helper). When
* `@opentelemetry/api` is installed, the tracer is `trace.getTracer(name)` and
* `contextWithSpan(span)` returns `trace.setSpan(context.active(), span)`. The
* `api` package returns a no-op tracer while no SDK is registered, so an unset
* endpoint still yields a safe no-op. The `api` package loads lazily through
* `require`, so the module graph stays OTel-free until the first call. The
* accessor never throws: a load or lookup failure logs once and returns the
* local no-op trace context (fail open).
*/
export function getStartupTraceContext(name = "paperclip.startup"): StartupTraceContextHandle {
try {
const require = createRequire(import.meta.url);
const api = require("@opentelemetry/api") as {
trace?: {
getTracer(n: string): StartupTracerHandle;
setSpan(context: unknown, span: unknown): unknown;
};
context?: { active(): unknown };
};
const trace = api.trace;
const context = api.context;
if (!trace?.getTracer || !trace.setSpan || !context?.active) {
return NOOP_TRACE_CONTEXT;
}
const tracer = trace.getTracer(name);
return {
tracer,
// Keep the method calls on `trace` / `context` so the api singletons stay
// their own receiver.
contextWithSpan: (span: unknown) => trace.setSpan(context.active(), span),
};
} catch (err) {
if (!traceContextApiLoadFailed) {
traceContextApiLoadFailed = true;
// eslint-disable-next-line no-console
console.warn(
"[paperclip] @opentelemetry/api is not available; startup tracing uses a no-op trace context.",
err,
);
}
return NOOP_TRACE_CONTEXT;
}
}
/**
* Resolves once the OTel SDK has started (or once bootstrap has failed and
* logged, or immediately when the feature is off). Await before constructing

56
server/src/services/environment-execution-target.ts
View File

@ -5,12 +5,37 @@ import {
adapterExecutionTargetToRemoteSpec,
type AdapterExecutionTarget,
} from "@paperclipai/adapter-utils/execution-target";
import { normalizeProviderFamily } from "@paperclipai/adapter-utils/acpx-engine/startup-timing";
import { parseObject } from "../adapters/utils.js";
import { getStartupTracer } from "../instrumentation.js";
import { resolveEnvironmentDriverConfigForRuntime } from "./environment-config.js";
import type { EnvironmentRuntimeService } from "./environment-runtime.js";
export const DEFAULT_SANDBOX_REMOTE_CWD = "/tmp";
/** The minimal span surface the provider-exec seam calls. A real injected OTel
* span satisfies it; the no-op tracer's span satisfies it too. */
type ExecSpan = {
setAttribute(key: string, value: string | number | boolean): void;
end(): void;
};
/** The minimal tracer surface the provider-exec seam calls. `getStartupTracer`
* returns a real or no-op implementation that satisfies it. */
type ExecTracer = { startSpan(name: string): ExecSpan };
/**
* Set a numeric span attribute only when the value is a finite number. A value
* that is absent, `NaN`, or `Infinity` yields no attribute — never a misleading
* `0`. This mirrors the host counter guard below and the `adapter-utils`
* startup-step guard.
*/
function setFiniteNumberAttr(span: ExecSpan, key: string, value: unknown): void {
if (typeof value === "number" && Number.isFinite(value)) {
span.setAttribute(key, value);
}
}
export async function resolveEnvironmentExecutionTarget(input: {
db: Db;
companyId: string;
@ -24,6 +49,10 @@ export async function resolveEnvironmentExecutionTarget(input: {
leaseMetadata: Record<string, unknown> | null;
lease?: EnvironmentLease | null;
environmentRuntime?: EnvironmentRuntimeService | null;
// The startup tracer for the provider-exec span. Defaults to the endpoint-
// gated server tracer, which is a no-op when tracing is off. Tests inject a
// recording tracer.
tracer?: ExecTracer;
}): Promise<AdapterExecutionTarget | null> {
if (input.environment.driver === "local") {
return {
@ -74,6 +103,13 @@ export async function resolveEnvironmentExecutionTarget(input: {
if (typeof get === "number" && Number.isFinite(get)) providerGetMs += get;
};
// The low-cardinality public provider family. A plugin-backed / operator-
// defined key maps to `plugin`, so a raw unbounded key never rides a span.
const providerFamily = normalizeProviderFamily(parsed.config.provider);
// The endpoint-gated startup tracer (no-op when tracing is off). Tests inject
// a recording tracer.
const tracer = input.tracer ?? getStartupTracer();
return {
kind: "remote",
transport: "sandbox",
@ -117,6 +153,26 @@ export async function resolveEnvironmentExecutionTarget(input: {
timeoutMs: commandInput.timeoutMs,
});
accumulateProviderDurations(result.metadata);
// Emit one span for this host→sandbox exec (opt-in; a no-op tracer
// when tracing is off). It carries ONLY closed-allowlist,
// low-cardinality attributes: the normalized provider family, the
// exit status (a two-value enum), and the host-received provider
// durations (per field, only when finite). The full command, args,
// env, stdout, and stderr never ride a span — not as an attribute
// and not as an event. The span name is the fixed operation label.
try {
const span = tracer.startSpan("provider.execute");
try {
span.setAttribute("provider", providerFamily);
span.setAttribute("exit", result.exitCode === 0 ? "ok" : "error");
setFiniteNumberAttr(span, "provider.exec.duration_ms", result.metadata?.durationMs);
setFiniteNumberAttr(span, "provider.get.duration_ms", result.metadata?.getDurationMs);
} finally {
span.end();
}
} catch {
// Observability must not change execution control flow.
}
if (result.stdout) await commandInput.onLog?.("stdout", result.stdout);
if (result.stderr) await commandInput.onLog?.("stderr", result.stderr);
return {

6
server/src/services/heartbeat.ts
View File

@ -68,6 +68,7 @@ import {
workspaceOperations,
} from "@paperclipai/db";
import { conflict, HttpError, notFound } from "../errors.js";
import { getStartupTraceContext } from "../instrumentation.js";
import { logger } from "../middleware/logger.js";
import { publishLiveEvent } from "./live-events.js";
import { normalizeResponsibleUserDenialCode } from "./responsible-user-denial-run-outcomes.js";
@ -14458,6 +14459,11 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
onLog,
onMeta: onAdapterMeta,
onEvent: onAdapterEvent,
// The endpoint-gated OpenTelemetry startup trace context. It is a
// no-op unless `OTEL_EXPORTER_OTLP_ENDPOINT` is set and the OTel
// packages are installed, so the sandbox-start span path stays inert
// by default.
startupTraceContext: getStartupTraceContext(),
onRuntimeProgress: async (progress) => {
await recordCurrentHeartbeatRunRuntimeProgress(run, progress, issueId);
},