From 8a93a0de4cba3ee37d564c83f58d6b7732b3cd62 Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Wed, 1 Jul 2026 11:39:39 -0700 Subject: [PATCH] Implement generated client telemetry types (#8818) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Telemetry is part of the control plane's operational visibility and needs stable event contracts. > - The shared telemetry client accepted first-party event names through a broad string surface, which weakened compile-time guarantees. > - Plugin telemetry still needs a dynamic path because plugin-defined events cannot be enumerated in the core generated type module. > - This pull request vendors generated Paperclip telemetry event and dimension types, closes the first-party event-name union, and keeps plugin telemetry on an explicit dynamic method. > - Review feedback clarified that backend normalization should remain the source of truth, so telemetry helpers now preserve raw categorical values while keeping generated per-event type hints. > - The benefit is stricter first-party telemetry typing without hiding backend normalization signals or changing batching, flushing, schema versioning, sinks, or endpoints. ## Linked Issues or Issue Description No public GitHub issue exists for this internal type-contract maintenance change. ### Problem or motivation The shared telemetry client should reject unregistered first-party event names at compile time, while the plugin telemetry bridge must continue to emit plugin-defined events through the existing batching and envelope path. Helper wrappers should also avoid client-side enum coercion so the backend can detect and record normalization when clients send unexpected categorical values. ### Proposed solution Generate and vendor the accepted Paperclip telemetry event and dimension types, use those types for the first-party `track()` API, keep plugin-defined telemetry on an explicit dynamic method, and let helper wrappers pass raw categorical dimensions through to backend validation. ### Alternatives considered Keeping `track()` open to arbitrary strings would preserve flexibility, but it would not give first-party callers the type safety this change is meant to provide. Enumerating plugin events in core was also ruled out because plugin-defined events are not known to the core package. Client-side enum normalization was removed after review because it duplicates backend validation and can hide misbehaving-client signals. ### Roadmap alignment This is a tightly scoped telemetry contract maintenance change and does not overlap with a roadmap-level core feature. ## What Changed - Vendored the generated Paperclip telemetry event and dimension type module under shared telemetry code. - Closed the first-party telemetry event-name union to generated backend-accepted names plus an explicit `RegisteredPluginEventName = never` extension point. - Added `TelemetryClient.trackDynamic()` for plugin telemetry bridge emission while keeping `track()` closed and typed. - Added JSDoc explaining when to use `track()` versus `trackDynamic()`. - Updated telemetry helper wrappers to type dimensions from each event's generated schema entry while passing raw categorical values through for backend normalization. - Added `trackInteractionResolved()` and updated focused shared/server tests for telemetry event typing, raw pass-through behavior, and plugin telemetry bridging. ## Verification Local verification passed before the latest push: - `pnpm --filter @paperclipai/shared typecheck` - `pnpm --filter @paperclipai/server typecheck` - `pnpm exec vitest run packages/shared/src/telemetry/client-types.test.ts server/src/__tests__/shared-telemetry-events.test.ts server/src/__tests__/plugin-telemetry-bridge.test.ts server/src/__tests__/project-goal-telemetry-routes.test.ts server/src/__tests__/routine-run-telemetry.test.ts server/src/__tests__/issue-telemetry-routes.test.ts` - `git diff --check` Post-push verification completed on head `3d973ffbea6154b19ad208dcffd1374d1b25b654`: - GitHub PR checks passed, including `verify`, build, typecheck/release registry, general test shards, serialized server shards, canary dry run, e2e, and security checks. - Greptile Review passed with 5/5 confidence. - All PR review threads are resolved. ## Risks Low runtime risk. The change is intended to affect TypeScript contracts and helper typing while preserving the existing telemetry enqueue, batching, and backend ingest path. The main intentional behavior shift is that helper wrappers no longer coerce unexpected categorical values on the client; those values reach the backend so backend normalization can record the signal. Private company import source refs still use `hashPrivateRef` when `isPrivate` is true. ## Model Used OpenAI GPT-5 Codex, tool-enabled coding agent. Exact context window was not exposed by the runtime; the agent used repository file access, shell commands, and GitHub CLI operations. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .../shared/src/telemetry/client-types.test.ts | 25 ++ packages/shared/src/telemetry/client.ts | 29 +- packages/shared/src/telemetry/events.ts | 115 ++++-- .../generated/paperclip-telemetry.ts | 371 ++++++++++++++++++ packages/shared/src/telemetry/index.ts | 10 + packages/shared/src/telemetry/types.ts | 29 +- .../__tests__/plugin-telemetry-bridge.test.ts | 14 +- .../project-goal-telemetry-routes.test.ts | 2 +- .../__tests__/shared-telemetry-events.test.ts | 61 ++- server/src/services/plugin-host-services.ts | 2 +- 10 files changed, 608 insertions(+), 50 deletions(-) create mode 100644 packages/shared/src/telemetry/client-types.test.ts create mode 100644 packages/shared/src/telemetry/generated/paperclip-telemetry.ts diff --git a/packages/shared/src/telemetry/client-types.test.ts b/packages/shared/src/telemetry/client-types.test.ts new file mode 100644 index 0000000000..3b630edda4 --- /dev/null +++ b/packages/shared/src/telemetry/client-types.test.ts @@ -0,0 +1,25 @@ +import { describe, expect, it } from "vitest"; +import { TelemetryClient } from "./client.js"; + +describe("TelemetryClient event-name types", () => { + it("keeps dynamic telemetry separate from registered events", () => { + const client = new TelemetryClient( + { enabled: false }, + () => ({ + installId: "test-install", + salt: "test-salt", + createdAt: "2026-01-01T00:00:00Z", + firstSeenVersion: "0.0.0", + }), + "0.0.0-test", + ); + + client.track("install.started", {}); + client.trackDynamic("plugin.linear.sync_completed", {}); + + // @ts-expect-error plugin events are intentionally not registered first-party events. + client.track("plugin.linear.sync_completed", {}); + + expect(client).toBeInstanceOf(TelemetryClient); + }); +}); diff --git a/packages/shared/src/telemetry/client.ts b/packages/shared/src/telemetry/client.ts index 9c4181cd3a..178e5f0556 100644 --- a/packages/shared/src/telemetry/client.ts +++ b/packages/shared/src/telemetry/client.ts @@ -1,7 +1,9 @@ import { createHash } from "node:crypto"; import type { TelemetryConfig, + TelemetryDimensions, TelemetryEvent, + TelemetryEventDimensions, TelemetryEventName, TelemetryState, } from "./types.js"; @@ -13,6 +15,11 @@ const DEFAULT_ENDPOINTS = [ const BATCH_SIZE = 50; const SEND_TIMEOUT_MS = 5_000; +type TrackArgs = + keyof TelemetryEventDimensions extends never + ? [dimensions?: TelemetryEventDimensions] + : [dimensions: TelemetryEventDimensions]; + export class TelemetryClient { private queue: TelemetryEvent[] = []; private readonly config: TelemetryConfig; @@ -27,14 +34,32 @@ export class TelemetryClient { this.version = version; } - track(eventName: TelemetryEventName, dimensions?: Record): void { + /** + * Tracks first-party Paperclip telemetry events registered in the generated + * backend event schema. + */ + track(eventName: K, ...args: TrackArgs): void { + const [dimensions] = args; + this.enqueue(eventName, dimensions); + } + + /** + * Tracks plugin telemetry bridge events whose names are built dynamically + * from third-party plugin input. The backend accepts only explicitly + * registered plugin events. + */ + trackDynamic(eventName: string, dimensions?: TelemetryDimensions): void { + this.enqueue(eventName, dimensions); + } + + private enqueue(eventName: string, dimensions?: object): void { if (!this.config.enabled) return; this.getState(); // ensure state is initialised (side-effect: creates state file on first call) this.queue.push({ name: eventName, occurredAt: new Date().toISOString(), - dimensions: dimensions ?? {}, + dimensions: { ...dimensions } as TelemetryDimensions, }); if (this.queue.length >= BATCH_SIZE) { diff --git a/packages/shared/src/telemetry/events.ts b/packages/shared/src/telemetry/events.ts index 1b347f3cd1..15315719b5 100644 --- a/packages/shared/src/telemetry/events.ts +++ b/packages/shared/src/telemetry/events.ts @@ -1,91 +1,123 @@ import type { TelemetryClient } from "./client.js"; +import type { EventDimensionsMap } from "./generated/paperclip-telemetry.js"; + +type RawDimension = T | (string & {}); + +function asEventDimension(value: RawDimension): T { + return value as T; +} export function trackInstallStarted(client: TelemetryClient): void { - client.track("install.started"); + client.track("install.started", {}); } export function trackInstallCompleted( client: TelemetryClient, - dims: { adapterType: string }, + dims: { adapterType: RawDimension }, ): void { - client.track("install.completed", { adapter_type: dims.adapterType }); + client.track("install.completed", { + adapter_type: asEventDimension(dims.adapterType), + }); } export function trackCompanyImported( client: TelemetryClient, - dims: { sourceType: string; sourceRef: string; isPrivate: boolean }, + dims: { + sourceType: RawDimension; + sourceRef: string; + isPrivate: boolean; + }, ): void { const ref = dims.isPrivate ? client.hashPrivateRef(dims.sourceRef) : dims.sourceRef; client.track("company.imported", { - source_type: dims.sourceType, + source_type: asEventDimension(dims.sourceType), source_ref: ref, source_ref_hashed: dims.isPrivate, }); } export function trackProjectCreated(client: TelemetryClient): void { - client.track("project.created"); + client.track("project.created", {}); } export function trackRoutineCreated(client: TelemetryClient): void { - client.track("routine.created"); + client.track("routine.created", {}); } export function trackRoutineRun( client: TelemetryClient, - dims: { source: string; status: string }, + dims: { + source: RawDimension; + status: RawDimension; + }, ): void { client.track("routine.run", { - source: dims.source, - status: dims.status, + source: asEventDimension(dims.source), + status: asEventDimension(dims.status), }); } export function trackGoalCreated( client: TelemetryClient, - dims?: { goalLevel?: string | null }, + dims?: { goalLevel?: RawDimension | null }, ): void { - client.track("goal.created", dims?.goalLevel ? { goal_level: dims.goalLevel } : undefined); + client.track("goal.created", { + goal_level: dims?.goalLevel ? asEventDimension(dims.goalLevel) : "other", + }); } export function trackAgentCreated( client: TelemetryClient, - dims: { agentRole: string; agentId?: string }, + dims: { + agentRole: RawDimension; + agentId: string; + }, ): void { client.track("agent.created", { - agent_role: dims.agentRole, - ...(dims.agentId ? { agent_id: dims.agentId } : {}), + agent_role: asEventDimension(dims.agentRole), + agent_id: dims.agentId, }); } export function trackSkillImported( client: TelemetryClient, - dims: { sourceType: string; skillRef?: string | null }, + dims: { + sourceType: RawDimension; + skillRef?: string | null; + }, ): void { client.track("skill.imported", { - source_type: dims.sourceType, + source_type: asEventDimension(dims.sourceType), ...(dims.skillRef ? { skill_ref: dims.skillRef } : {}), }); } export function trackAgentFirstHeartbeat( client: TelemetryClient, - dims: { agentRole: string; agentId?: string }, + dims: { + agentRole: RawDimension; + agentId: string; + }, ): void { client.track("agent.first_heartbeat", { - agent_role: dims.agentRole, - ...(dims.agentId ? { agent_id: dims.agentId } : {}), + agent_role: asEventDimension(dims.agentRole), + agent_id: dims.agentId, }); } export function trackAgentTaskCompleted( client: TelemetryClient, - dims: { agentRole: string; agentId?: string; adapterType?: string; model?: string }, + dims: { + agentRole: RawDimension; + agentId: string; + adapterType: RawDimension; + model?: string; + }, ): void { client.track("agent.task_completed", { - agent_role: dims.agentRole, - ...(dims.agentId ? { agent_id: dims.agentId } : {}), - ...(dims.adapterType ? { adapter_type: dims.adapterType } : {}), + agent_role: asEventDimension(dims.agentRole), + agent_id: dims.agentId, + adapter_type: asEventDimension(dims.adapterType), ...(dims.model ? { model: dims.model } : {}), }); } @@ -96,3 +128,38 @@ export function trackErrorHandlerCrash( ): void { client.track("error.handler_crash", { error_code: dims.errorCode }); } + +export function trackInteractionResolved( + client: TelemetryClient, + dims: { + interactionKind: RawDimension; + status: RawDimension; + resolvedByKind: RawDimension; + resolutionReason?: RawDimension | null; + createdByKind?: RawDimension | null; + creatorAgentRole?: RawDimension | null; + continuationPolicy?: RawDimension | null; + targetType?: RawDimension | null; + optionCount?: number; + selectedOptionCount?: number; + questionCount?: number; + answeredQuestionCount?: number; + createdTaskCount?: number; + }, +): void { + client.track("interaction.resolved", { + interaction_kind: asEventDimension(dims.interactionKind), + status: asEventDimension(dims.status), + resolved_by_kind: asEventDimension(dims.resolvedByKind), + ...(dims.resolutionReason ? { resolution_reason: asEventDimension(dims.resolutionReason) } : {}), + ...(dims.createdByKind ? { created_by_kind: asEventDimension(dims.createdByKind) } : {}), + ...(dims.creatorAgentRole ? { creator_agent_role: asEventDimension(dims.creatorAgentRole) } : {}), + ...(dims.continuationPolicy ? { continuation_policy: asEventDimension(dims.continuationPolicy) } : {}), + ...(dims.targetType ? { target_type: asEventDimension(dims.targetType) } : {}), + ...(dims.optionCount === undefined ? {} : { option_count: dims.optionCount }), + ...(dims.selectedOptionCount === undefined ? {} : { selected_option_count: dims.selectedOptionCount }), + ...(dims.questionCount === undefined ? {} : { question_count: dims.questionCount }), + ...(dims.answeredQuestionCount === undefined ? {} : { answered_question_count: dims.answeredQuestionCount }), + ...(dims.createdTaskCount === undefined ? {} : { created_task_count: dims.createdTaskCount }), + }); +} diff --git a/packages/shared/src/telemetry/generated/paperclip-telemetry.ts b/packages/shared/src/telemetry/generated/paperclip-telemetry.ts new file mode 100644 index 0000000000..05ef53b9c4 --- /dev/null +++ b/packages/shared/src/telemetry/generated/paperclip-telemetry.ts @@ -0,0 +1,371 @@ +// GENERATED — DO NOT EDIT. + +export interface PaperclipAgentCreatedDimensions { +agent_id: string +agent_role: ("ceo" | "cto" | "cmo" | "cfo" | "security" | "engineer" | "designer" | "pm" | "qa" | "devops" | "researcher" | "general" | "other") +} + +export interface PaperclipAgentFirstHeartbeatDimensions { +agent_id: string +agent_role: ("ceo" | "cto" | "cmo" | "cfo" | "security" | "engineer" | "designer" | "pm" | "qa" | "devops" | "researcher" | "general" | "other") +} + +export interface PaperclipAgentTaskCompletedDimensions { +adapter_type: ("process" | "http" | "acpx_local" | "claude_local" | "codex_local" | "cursor_cloud" | "gemini_local" | "hermes_gateway" | "hermes_local" | "opencode_local" | "pi_local" | "cursor" | "openclaw_gateway" | "grok_local" | "other") +agent_id: string +agent_role: ("ceo" | "cto" | "cmo" | "cfo" | "security" | "engineer" | "designer" | "pm" | "qa" | "devops" | "researcher" | "general" | "other") +model?: string +} + +export interface PaperclipCompanyImportedDimensions { +source_type: ("local_path" | "github" | "url" | "catalog" | "skills_sh" | "unknown") +source_ref?: string +source_ref_hashed?: boolean +} + +export interface PaperclipErrorHandlerCrashDimensions { +error_code: string +} + +export interface PaperclipGoalCreatedDimensions { +goal_level: ("company" | "team" | "agent" | "task" | "other") +} + +export interface PaperclipInstallCompletedDimensions { +adapter_type: ("process" | "http" | "acpx_local" | "claude_local" | "codex_local" | "cursor_cloud" | "gemini_local" | "hermes_gateway" | "hermes_local" | "opencode_local" | "pi_local" | "cursor" | "openclaw_gateway" | "grok_local" | "other") +} + +export interface PaperclipInstallStartedDimensions { + +} + +export interface PaperclipInteractionResolvedDimensions { +interaction_kind: ("suggest_tasks" | "ask_user_questions" | "request_confirmation" | "request_checkbox_confirmation" | "other") +status: ("accepted" | "rejected" | "answered" | "cancelled" | "expired" | "failed" | "other") +resolution_reason?: ("accepted" | "rejected" | "stale_target" | "superseded_by_comment" | "expired" | "cancelled" | "other") +resolved_by_kind: ("user" | "agent" | "system" | "other") +created_by_kind?: ("agent" | "user" | "other") +creator_agent_role?: ("ceo" | "cto" | "cmo" | "cfo" | "security" | "engineer" | "designer" | "pm" | "qa" | "devops" | "researcher" | "general" | "other") +continuation_policy?: ("none" | "wake_assignee" | "wake_assignee_on_accept" | "other") +target_type?: ("issue_document" | "custom" | "none" | "other") +option_count?: number +selected_option_count?: number +question_count?: number +answered_question_count?: number +created_task_count?: number +skipped_task_count?: number +has_reason?: boolean +resolution_latency_seconds?: number +interaction_id?: string +created_by_agent_id?: string +source_run_id?: string +} + +export interface PaperclipProjectCreatedDimensions { + +} + +export interface PaperclipRoutineCreatedDimensions { + +} + +export interface PaperclipRoutineRunDimensions { +source: ("schedule" | "manual" | "api" | "webhook" | "other") +status: ("received" | "coalesced" | "skipped" | "issue_created" | "completed" | "failed" | "other") +} + +export interface PaperclipSkillImportedDimensions { +source_type: ("local_path" | "github" | "url" | "catalog" | "skills_sh" | "unknown") +skill_ref?: string +} + +export type PaperclipEventName = + | "agent.created" + | "agent.first_heartbeat" + | "agent.task_completed" + | "company.imported" + | "error.handler_crash" + | "goal.created" + | "install.completed" + | "install.started" + | "interaction.resolved" + | "project.created" + | "routine.created" + | "routine.run" + | "skill.imported"; + +export interface EventDimensionsMap { + "agent.created": PaperclipAgentCreatedDimensions; + "agent.first_heartbeat": PaperclipAgentFirstHeartbeatDimensions; + "agent.task_completed": PaperclipAgentTaskCompletedDimensions; + "company.imported": PaperclipCompanyImportedDimensions; + "error.handler_crash": PaperclipErrorHandlerCrashDimensions; + "goal.created": PaperclipGoalCreatedDimensions; + "install.completed": PaperclipInstallCompletedDimensions; + "install.started": PaperclipInstallStartedDimensions; + "interaction.resolved": PaperclipInteractionResolvedDimensions; + "project.created": PaperclipProjectCreatedDimensions; + "routine.created": PaperclipRoutineCreatedDimensions; + "routine.run": PaperclipRoutineRunDimensions; + "skill.imported": PaperclipSkillImportedDimensions; +} + +export const PAPERCLIP_EVENTS = { + "agent.created": "agent.created", + "agent.first_heartbeat": "agent.first_heartbeat", + "agent.task_completed": "agent.task_completed", + "company.imported": "company.imported", + "error.handler_crash": "error.handler_crash", + "goal.created": "goal.created", + "install.completed": "install.completed", + "install.started": "install.started", + "interaction.resolved": "interaction.resolved", + "project.created": "project.created", + "routine.created": "routine.created", + "routine.run": "routine.run", + "skill.imported": "skill.imported", +} as const; + +export const PAPERCLIP_ENUM_DESCRIPTIONS = { + "agent.created": { + "agent_role": { + "ceo": "Agent configured for company leadership and board coordination work.", + "cto": "Agent configured for technical leadership, architecture, and engineering coordination.", + "cmo": "Agent configured for marketing leadership work.", + "cfo": "Agent configured for finance leadership work.", + "security": "Agent configured for security review, risk, or policy work.", + "engineer": "Agent configured for general software engineering work.", + "designer": "Agent configured for product, visual, or experience design work.", + "pm": "Agent configured for product management or planning work.", + "qa": "Agent configured for quality assurance, testing, or validation work.", + "devops": "Agent configured for infrastructure, deployment, or operations work.", + "researcher": "Agent configured for research and information-gathering work.", + "general": "Agent configured as a general-purpose worker without a more specific role.", + "other": "Fallback when the agent role is unknown or not represented by the tracked enum." + } + }, + "agent.first_heartbeat": { + "agent_role": { + "ceo": "Agent configured for company leadership and board coordination work.", + "cto": "Agent configured for technical leadership, architecture, and engineering coordination.", + "cmo": "Agent configured for marketing leadership work.", + "cfo": "Agent configured for finance leadership work.", + "security": "Agent configured for security review, risk, or policy work.", + "engineer": "Agent configured for general software engineering work.", + "designer": "Agent configured for product, visual, or experience design work.", + "pm": "Agent configured for product management or planning work.", + "qa": "Agent configured for quality assurance, testing, or validation work.", + "devops": "Agent configured for infrastructure, deployment, or operations work.", + "researcher": "Agent configured for research and information-gathering work.", + "general": "Agent configured as a general-purpose worker without a more specific role.", + "other": "Fallback when the agent role is unknown or not represented by the tracked enum." + } + }, + "agent.task_completed": { + "adapter_type": { + "process": "Agent runtime uses a local process adapter.", + "http": "Agent runtime uses a generic HTTP adapter.", + "acpx_local": "Agent runtime uses the local ACPX adapter.", + "claude_local": "Agent runtime uses the local Claude adapter.", + "codex_local": "Agent runtime uses the local Codex adapter.", + "cursor_cloud": "Agent runtime uses the Cursor cloud adapter.", + "gemini_local": "Agent runtime uses the local Gemini adapter.", + "hermes_gateway": "Agent runtime uses the Hermes gateway adapter.", + "hermes_local": "Agent runtime uses the local Hermes adapter.", + "opencode_local": "Agent runtime uses the local OpenCode adapter.", + "pi_local": "Agent runtime uses the local Pi adapter.", + "cursor": "Agent runtime uses the Cursor adapter.", + "openclaw_gateway": "Agent runtime uses the OpenClaw gateway adapter.", + "grok_local": "Agent runtime uses the local Grok adapter.", + "other": "Fallback when the adapter type is unknown or not represented by the tracked enum." + }, + "agent_role": { + "ceo": "Agent configured for company leadership and board coordination work.", + "cto": "Agent configured for technical leadership, architecture, and engineering coordination.", + "cmo": "Agent configured for marketing leadership work.", + "cfo": "Agent configured for finance leadership work.", + "security": "Agent configured for security review, risk, or policy work.", + "engineer": "Agent configured for general software engineering work.", + "designer": "Agent configured for product, visual, or experience design work.", + "pm": "Agent configured for product management or planning work.", + "qa": "Agent configured for quality assurance, testing, or validation work.", + "devops": "Agent configured for infrastructure, deployment, or operations work.", + "researcher": "Agent configured for research and information-gathering work.", + "general": "Agent configured as a general-purpose worker without a more specific role.", + "other": "Fallback when the agent role is unknown or not represented by the tracked enum." + } + }, + "company.imported": { + "source_type": { + "local_path": "Import source came from a filesystem path on the operator's machine.", + "github": "Import source came from a GitHub repository or GitHub-backed reference.", + "url": "Import source came from a direct URL.", + "catalog": "Import source came from a Paperclip catalog entry.", + "skills_sh": "Import source came from a Skills.sh-compatible source.", + "unknown": "Source type could not be classified." + } + }, + "goal.created": { + "goal_level": { + "company": "Goal applies at company scope.", + "team": "Goal applies at team or group scope.", + "agent": "Goal applies to a specific agent.", + "task": "Goal applies to task-level work.", + "other": "Fallback when the goal level is unknown or not represented by the tracked enum." + } + }, + "install.completed": { + "adapter_type": { + "process": "Agent runtime uses a local process adapter.", + "http": "Agent runtime uses a generic HTTP adapter.", + "acpx_local": "Agent runtime uses the local ACPX adapter.", + "claude_local": "Agent runtime uses the local Claude adapter.", + "codex_local": "Agent runtime uses the local Codex adapter.", + "cursor_cloud": "Agent runtime uses the Cursor cloud adapter.", + "gemini_local": "Agent runtime uses the local Gemini adapter.", + "hermes_gateway": "Agent runtime uses the Hermes gateway adapter.", + "hermes_local": "Agent runtime uses the local Hermes adapter.", + "opencode_local": "Agent runtime uses the local OpenCode adapter.", + "pi_local": "Agent runtime uses the local Pi adapter.", + "cursor": "Agent runtime uses the Cursor adapter.", + "openclaw_gateway": "Agent runtime uses the OpenClaw gateway adapter.", + "grok_local": "Agent runtime uses the local Grok adapter.", + "other": "Fallback when the adapter type is unknown or not represented by the tracked enum." + } + }, + "interaction.resolved": { + "interaction_kind": { + "suggest_tasks": "Board-facing interaction that proposes concrete subtasks for acceptance.", + "ask_user_questions": "Board-facing interaction that asks structured questions and stores answers.", + "request_confirmation": "Board-facing interaction that asks for a single accept or reject decision.", + "request_checkbox_confirmation": "Board-facing interaction that asks the board to select options and confirm.", + "other": "Fallback when the interaction kind is unknown or not represented by the tracked enum." + }, + "status": { + "accepted": "Interaction was accepted.", + "rejected": "Interaction was rejected.", + "answered": "Ask-user-questions interaction was answered.", + "cancelled": "Interaction was cancelled before acceptance or answer.", + "expired": "Interaction expired because the bound target became stale or was superseded.", + "failed": "Interaction resolution attempted but failed.", + "other": "Fallback when the terminal status is unknown or not represented by the tracked enum." + }, + "resolution_reason": { + "accepted": "Stored result outcome says the interaction was accepted.", + "rejected": "Stored result outcome says the interaction was rejected.", + "stale_target": "Bound target, such as an issue document revision, was no longer current.", + "superseded_by_comment": "A later user or board comment superseded the pending confirmation.", + "expired": "Interaction expired for a generic expiration reason.", + "cancelled": "Interaction was explicitly cancelled.", + "other": "Fallback when the resolution reason is unknown or not represented by the tracked enum." + }, + "resolved_by_kind": { + "user": "A board or human user resolved the interaction.", + "agent": "An agent resolved the interaction.", + "system": "Paperclip resolved the interaction automatically.", + "other": "Fallback when the resolver kind is unknown or not represented by the tracked enum." + }, + "created_by_kind": { + "agent": "Interaction was created by an agent.", + "user": "Interaction was created by a board or human user.", + "other": "Fallback when the creator kind is unknown or not represented by the tracked enum." + }, + "creator_agent_role": { + "ceo": "Agent configured for company leadership and board coordination work.", + "cto": "Agent configured for technical leadership, architecture, and engineering coordination.", + "cmo": "Agent configured for marketing leadership work.", + "cfo": "Agent configured for finance leadership work.", + "security": "Agent configured for security review, risk, or policy work.", + "engineer": "Agent configured for general software engineering work.", + "designer": "Agent configured for product, visual, or experience design work.", + "pm": "Agent configured for product management or planning work.", + "qa": "Agent configured for quality assurance, testing, or validation work.", + "devops": "Agent configured for infrastructure, deployment, or operations work.", + "researcher": "Agent configured for research and information-gathering work.", + "general": "Agent configured as a general-purpose worker without a more specific role.", + "other": "Fallback when the agent role is unknown or not represented by the tracked enum." + }, + "continuation_policy": { + "none": "Resolving the interaction does not automatically wake the issue assignee.", + "wake_assignee": "Resolving the interaction wakes or returns the issue to the assignee.", + "wake_assignee_on_accept": "Accepting the interaction wakes or returns the issue to the assignee.", + "other": "Fallback when the continuation policy is unknown or not represented by the tracked enum." + }, + "target_type": { + "issue_document": "Interaction is bound to a specific issue document revision.", + "custom": "Interaction is bound to a custom target.", + "none": "Interaction has no bound target.", + "other": "Fallback when the target type is unknown or not represented by the tracked enum." + } + }, + "routine.run": { + "source": { + "schedule": "Routine was triggered by a scheduled trigger.", + "manual": "Routine was triggered manually by a user or agent action.", + "api": "Routine was triggered through an API request.", + "webhook": "Routine was triggered by a webhook.", + "other": "Fallback when the source is unknown or not represented by the tracked enum." + }, + "status": { + "received": "Routine run was accepted for processing.", + "coalesced": "A live execution already existed and the run was coalesced into it.", + "skipped": "A live execution already existed and concurrency policy skipped the run.", + "issue_created": "Routine dispatch created a new issue and queued the agent wakeup.", + "completed": "Routine run completed without needing a new issue.", + "failed": "Routine dispatch failed and the run was finalized as failed.", + "other": "Fallback when the status is unknown or not represented by the tracked enum." + } + }, + "skill.imported": { + "source_type": { + "local_path": "Import source came from a filesystem path on the operator's machine.", + "github": "Import source came from a GitHub repository or GitHub-backed reference.", + "url": "Import source came from a direct URL.", + "catalog": "Import source came from a Paperclip catalog entry.", + "skills_sh": "Import source came from a Skills.sh-compatible source.", + "unknown": "Source type could not be classified." + } + } +} as const; + +export const SCHEMA_VERSION = "1" as const; + +export interface PaperclipTelemetryEvent { +name: K +occurredAt: string +dimensions: EventDimensionsMap[K] +} + +export type AnyPaperclipTelemetryEvent = { + [K in PaperclipEventName]: PaperclipTelemetryEvent +}[PaperclipEventName]; + +export interface PaperclipTelemetryBatch { +app: "paperclip" +schemaVersion: typeof SCHEMA_VERSION +installId: string +version?: string +events: AnyPaperclipTelemetryEvent[] +} + +export function makeEvent( + name: K, + dimensions: EventDimensionsMap[K], + occurredAt: string +): PaperclipTelemetryEvent { + return { name, occurredAt, dimensions }; +} + +export function makeBatch( + installId: string, + events: readonly AnyPaperclipTelemetryEvent[], + version?: string +): PaperclipTelemetryBatch { + return { + app: "paperclip", + schemaVersion: SCHEMA_VERSION, + installId, + ...(version === undefined ? {} : { version }), + events: [...events] + }; +} diff --git a/packages/shared/src/telemetry/index.ts b/packages/shared/src/telemetry/index.ts index f80de29c3b..16b5d8de5d 100644 --- a/packages/shared/src/telemetry/index.ts +++ b/packages/shared/src/telemetry/index.ts @@ -14,11 +14,21 @@ export { trackAgentFirstHeartbeat, trackAgentTaskCompleted, trackErrorHandlerCrash, + trackInteractionResolved, } from "./events.js"; export type { TelemetryConfig, TelemetryState, TelemetryEvent, TelemetryEventEnvelope, + TelemetryDimensions, + TelemetryDimensionValue, + TelemetryEventDimensions, TelemetryEventName, + RegisteredPluginEventName, } from "./types.js"; +export type { + AnyPaperclipTelemetryEvent, + EventDimensionsMap, + PaperclipEventName, +} from "./generated/paperclip-telemetry.js"; diff --git a/packages/shared/src/telemetry/types.ts b/packages/shared/src/telemetry/types.ts index d3552d0d21..b5c2aef21b 100644 --- a/packages/shared/src/telemetry/types.ts +++ b/packages/shared/src/telemetry/types.ts @@ -1,3 +1,8 @@ +import type { + EventDimensionsMap, + PaperclipEventName, +} from "./generated/paperclip-telemetry.js"; + export interface TelemetryState { installId: string; salt: string; @@ -12,11 +17,14 @@ export interface TelemetryConfig { schemaVersion?: string; } +export type TelemetryDimensionValue = string | number | boolean; +export type TelemetryDimensions = Record; + /** Per-event object inside the backend envelope */ export interface TelemetryEvent { name: string; occurredAt: string; - dimensions: Record; + dimensions: TelemetryDimensions; } /** Full payload sent to the backend ingest endpoint */ @@ -28,17 +36,8 @@ export interface TelemetryEventEnvelope { events: TelemetryEvent[]; } -export type TelemetryEventName = - | "install.started" - | "install.completed" - | "company.imported" - | "project.created" - | "routine.created" - | "routine.run" - | "goal.created" - | "agent.created" - | "skill.imported" - | "agent.first_heartbeat" - | "agent.task_completed" - | "error.handler_crash" - | `plugin.${string}`; +export type RegisteredPluginEventName = never; +export type TelemetryEventName = PaperclipEventName | RegisteredPluginEventName; + +export type TelemetryEventDimensions = + K extends keyof EventDimensionsMap ? EventDimensionsMap[K] : never; diff --git a/server/src/__tests__/plugin-telemetry-bridge.test.ts b/server/src/__tests__/plugin-telemetry-bridge.test.ts index 367663d9bb..2b3b483627 100644 --- a/server/src/__tests__/plugin-telemetry-bridge.test.ts +++ b/server/src/__tests__/plugin-telemetry-bridge.test.ts @@ -26,8 +26,8 @@ describe("plugin telemetry bridge", () => { }); it("prefixes plugin telemetry events before forwarding them to the telemetry client", async () => { - const track = vi.fn(); - mockGetTelemetryClient.mockReturnValue({ track }); + const trackDynamic = vi.fn(); + mockGetTelemetryClient.mockReturnValue({ trackDynamic }); const services = buildHostServices( {} as never, @@ -46,14 +46,14 @@ describe("plugin telemetry bridge", () => { dimensions: { attempts: 2, success: true }, }); - expect(track).toHaveBeenCalledWith("plugin.linear.sync_completed", { + expect(trackDynamic).toHaveBeenCalledWith("plugin.linear.sync_completed", { attempts: 2, success: true, }); }); it("rejects invalid bare telemetry event names before prefixing", async () => { - mockGetTelemetryClient.mockReturnValue({ track: vi.fn() }); + mockGetTelemetryClient.mockReturnValue({ trackDynamic: vi.fn() }); const services = buildHostServices( {} as never, @@ -92,6 +92,9 @@ describe("plugin telemetry bridge", () => { }); it("passes telemetry requests through when the plugin declares the capability", async () => { + const trackDynamic = vi.fn(); + mockGetTelemetryClient.mockReturnValue({ trackDynamic }); + const services = buildHostServices( {} as never, "plugin-record-id", @@ -110,5 +113,8 @@ describe("plugin telemetry bridge", () => { }); expect(mockGetTelemetryClient).toHaveBeenCalledTimes(1); + expect(trackDynamic).toHaveBeenCalledWith("plugin.linear.sync_completed", { + source: "manual", + }); }); }); diff --git a/server/src/__tests__/project-goal-telemetry-routes.test.ts b/server/src/__tests__/project-goal-telemetry-routes.test.ts index 7b5b97bc1b..a0c12d799c 100644 --- a/server/src/__tests__/project-goal-telemetry-routes.test.ts +++ b/server/src/__tests__/project-goal-telemetry-routes.test.ts @@ -150,7 +150,7 @@ describe("project and goal telemetry routes", () => { .send({ name: "Telemetry project" }); expect([200, 201], JSON.stringify(res.body)).toContain(res.status); - expect(mockTelemetryTrack).toHaveBeenCalledWith("project.created"); + expect(mockTelemetryTrack).toHaveBeenCalledWith("project.created", {}); }); it("emits telemetry when a goal is created", async () => { diff --git a/server/src/__tests__/shared-telemetry-events.test.ts b/server/src/__tests__/shared-telemetry-events.test.ts index 1d6483d2e8..8150910ae9 100644 --- a/server/src/__tests__/shared-telemetry-events.test.ts +++ b/server/src/__tests__/shared-telemetry-events.test.ts @@ -3,9 +3,10 @@ import { trackAgentCreated, trackAgentFirstHeartbeat, trackAgentTaskCompleted, + trackInteractionResolved, trackInstallCompleted, } from "@paperclipai/shared/telemetry"; -import type { TelemetryClient } from "@paperclipai/shared/telemetry"; +import type { EventDimensionsMap, TelemetryClient } from "@paperclipai/shared/telemetry"; function createClient(): TelemetryClient { return { @@ -14,6 +15,10 @@ function createClient(): TelemetryClient { } as unknown as TelemetryClient; } +function runtimeValue(value: string): T { + return value as T; +} + describe("shared telemetry agent events", () => { it("includes agent_id for agent.created", () => { const client = createClient(); @@ -29,16 +34,30 @@ describe("shared telemetry agent events", () => { }); }); + it("passes an unrecognized agent role through for backend normalization", () => { + const client = createClient(); + + trackAgentCreated(client, { + agentRole: runtimeValue("coder"), + agentId: "44444444-4444-4444-8444-444444444444", + }); + + expect(client.track).toHaveBeenCalledWith("agent.created", { + agent_role: "coder", + agent_id: "44444444-4444-4444-8444-444444444444", + }); + }); + it("includes agent_id for agent.first_heartbeat", () => { const client = createClient(); trackAgentFirstHeartbeat(client, { - agentRole: "coder", + agentRole: "engineer", agentId: "22222222-2222-4222-8222-222222222222", }); expect(client.track).toHaveBeenCalledWith("agent.first_heartbeat", { - agent_role: "coder", + agent_role: "engineer", agent_id: "22222222-2222-4222-8222-222222222222", }); }); @@ -49,11 +68,13 @@ describe("shared telemetry agent events", () => { trackAgentTaskCompleted(client, { agentRole: "qa", agentId: "33333333-3333-4333-8333-333333333333", + adapterType: "codex_local", }); expect(client.track).toHaveBeenCalledWith("agent.task_completed", { agent_role: "qa", agent_id: "33333333-3333-4333-8333-333333333333", + adapter_type: "codex_local", }); }); @@ -70,4 +91,38 @@ describe("shared telemetry agent events", () => { expect.objectContaining({ agent_id: expect.any(String) }), ); }); + + it("passes interaction.resolved enum dimensions through for backend normalization", () => { + const client = createClient(); + + trackInteractionResolved(client, { + interactionKind: runtimeValue( + "single_confirmation", + ), + status: "accepted", + resolvedByKind: runtimeValue("operator"), + resolutionReason: "accepted", + createdByKind: "agent", + creatorAgentRole: runtimeValue("coder"), + continuationPolicy: runtimeValue( + "wake_everyone", + ), + targetType: "issue_document", + optionCount: 2, + selectedOptionCount: 1, + }); + + expect(client.track).toHaveBeenCalledWith("interaction.resolved", { + interaction_kind: "single_confirmation", + status: "accepted", + resolved_by_kind: "operator", + resolution_reason: "accepted", + created_by_kind: "agent", + creator_agent_role: "coder", + continuation_policy: "wake_everyone", + target_type: "issue_document", + option_count: 2, + selected_option_count: 1, + }); + }); }); diff --git a/server/src/services/plugin-host-services.ts b/server/src/services/plugin-host-services.ts index 4309e1e3c0..5284af6ee6 100644 --- a/server/src/services/plugin-host-services.ts +++ b/server/src/services/plugin-host-services.ts @@ -1294,7 +1294,7 @@ export function buildHostServices( } const telemetryClient = getTelemetryClient(); if (!telemetryClient) return; - telemetryClient.track(`plugin.${pluginKey}.${eventName}`, params.dimensions); + telemetryClient.trackDynamic(`plugin.${pluginKey}.${eventName}`, params.dimensions); }, },