From 90f85a7d11c517b1d09db90dbec97f4de7d96b83 Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Mon, 13 Jul 2026 23:47:27 -0700 Subject: [PATCH] Add telemetry proposal extractor (#9544) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - It emits telemetry events to understand product usage — registered event names are gated by a generated `PAPERCLIP_EVENTS` registry so the client only enqueues known, schema-approved events > - When product teams want to instrument a new behaviour, they must first register the event name — but schema registration is a commit-and-release cycle, which creates friction in fast-moving product iterations > - A proposal lane is needed: let developers mark a `track()` call with a typed `@ts-expect-error` proposal marker so the event name can be reviewed and tracked in CI before the schema is formally registered > - The existing client had no guard against unregistered event names, so any call with an out-of-registry name (or a prototype-inherited key) would silently enter the queue, state, and network flush path > - This PR adds an `Object.hasOwn(PAPERCLIP_EVENTS, eventName)` guard at the entry point of `track()` to swallow unregistered calls before any side effects, adds `scripts/extract-proposed-events.mjs` to scan source for proposal markers and emit a v2 JSON manifest with provenance and rationale, and documents the complete proposal workflow > - The benefit is that new instrumentation can be proposed and reviewed in code without touching the registered schema, and tooling can surface missing rationale before events graduate to stable ## Linked Issues or Issue Description No existing GitHub issue covers this change. This PR introduces a new feature. **Feature motivation:** Paperclip's telemetry schema is intentionally stable — registered event names are code-generated and gated. Product engineers who want to instrument a new behaviour today must land a schema change first, creating a two-step process that slows iteration. A proposal lane lets developers write the instrumentation call ahead of schema registration, protected by a compile-time `@ts-expect-error` marker that an extractor script can surface for review. This PR implements both the client-side safety gate and the extraction tooling. Refs: #9518 (closed predecessor — docs-only; this PR supersedes it with the full implementation) ## What Changed - Added `Object.hasOwn(PAPERCLIP_EVENTS, eventName)` guard at the top of `TelemetryClient.track()`: unregistered event names (including prototype-inherited keys) are now swallowed before any state, queue, or network operation - Added `scripts/extract-proposed-events.mjs`: scans TypeScript source for `@ts-expect-error -- proposed-telemetry(): ` markers; emits a v2 JSON manifest per proposed event including name, rationale, provenance (repo-relative file + line), and a `rationale_missing` flag for CI enforcement - Added `scripts/extract-proposed-events.test.mjs`: test suite covering marker parsing, multi-line markers, path validation, out-of-repo rejection, and the v2 schema output contract - Added `doc/TELEMETRY_WORKFLOW.md`: documents the proposal workflow, the canonical multi-line marker example, rationale requirements, and how to graduate a proposed event to stable schema - Updated `packages/shared/src/telemetry/README.md`: added "Proposed Events" section to the Telemetry Data Contract per the contributing guide requirement for telemetry changes ## Verification Run all of the following from the repo root: ```sh # Extractor unit tests node --test scripts/extract-proposed-events.test.mjs # Telemetry client + types tests pnpm exec vitest run --config vitest.config.ts \ src/telemetry/client.test.ts src/telemetry/client-types.test.ts \ --reporter=verbose # (run from packages/shared) # Type-check pnpm --filter @paperclipai/shared typecheck # Smoke-run the extractor in local-test mode node scripts/extract-proposed-events.mjs --ref local-test ``` All four commands pass locally. ## Risks - **Silent drop on unregistered events:** The `Object.hasOwn` guard fails closed — any event name not in `PAPERCLIP_EVENTS` is silently dropped. If the generated registry is missing an event that was previously tracked, those calls will be silently lost. Mitigation: the extractor script surfaces proposed events that need registration; the TypeScript type system already enforces `TelemetryEventName ⊆ PAPERCLIP_EVENTS` at compile time. - **Extractor is read-only:** `extract-proposed-events.mjs` reads source and emits JSON; it does not modify any files. No runtime or schema risk. - Overall risk: **low**. The guard is additive and defensive; the extractor and docs are additive only. ## Model Used - Provider: Anthropic - Model ID: `claude-sonnet-4-6` - Context window: 200 K tokens - Capabilities: tool use, extended context, code generation ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/TELEMETRY_WORKFLOW.md | 59 +++ packages/shared/src/telemetry/README.md | 14 +- packages/shared/src/telemetry/client.test.ts | 103 ++++++ packages/shared/src/telemetry/client.ts | 2 + scripts/extract-proposed-events.mjs | 358 +++++++++++++++++++ scripts/extract-proposed-events.test.mjs | 261 ++++++++++++++ 6 files changed, 793 insertions(+), 4 deletions(-) create mode 100644 doc/TELEMETRY_WORKFLOW.md create mode 100644 packages/shared/src/telemetry/client.test.ts create mode 100644 scripts/extract-proposed-events.mjs create mode 100644 scripts/extract-proposed-events.test.mjs diff --git a/doc/TELEMETRY_WORKFLOW.md b/doc/TELEMETRY_WORKFLOW.md new file mode 100644 index 0000000000..734442291a --- /dev/null +++ b/doc/TELEMETRY_WORKFLOW.md @@ -0,0 +1,59 @@ +# Telemetry Workflow + +Paperclip first-party telemetry is schema-led for stable events and proposal-led for new product instrumentation. + +Stable events must be present in `packages/shared/src/telemetry/generated/paperclip-telemetry.ts` before normal client code emits them. Proposed events may be added ahead of schema registration only with an `@ts-expect-error` proposal marker on the `client.track()` event-name argument. + +## Proposed Events + +A proposed event is a normal `client.track()` call whose event name is not yet in `PaperclipEventName`. The runtime client swallows unregistered first-party event names before queueing, state initialization, or network flush, so proposed events do not leave the process until the generated telemetry schema adopts the event name. + +Use this marker shape when possible: + +```ts +import type { TelemetryClient } from "./client.js"; + +export function trackYourFeatureActionPerformed( + client: TelemetryClient, + dims: { + action_source: "toolbar" | "menu" | (string & {}); + item_count: number; + }, +): void { + client.track( + // @ts-expect-error -- proposed-telemetry(https://github.com/paperclipai/paperclip/issues/123): measure feature action completion + "your_feature.action_performed", + dims, + ); +} +``` + +The multi-line shape is recommended because TypeScript places the unregistered-name error on the event-name line. When the schema later registers the event, that error disappears and TypeScript raises TS2578 for the now-unused directive, which tells the adopter to remove the marker in the same change that syncs the generated schema. + +The suffix format is: + +```text +-- proposed-telemetry(): +``` + +`` should be a public `https://github.com/paperclipai/paperclip/issues/123` URL. The rationale should be a short product reason for collecting the event. Missing issue or rationale text is tolerated at the call site and flagged by `scripts/extract-proposed-events.mjs`; it is not an OSS CI failure. + +These formatting conventions are documentation-only. Do not add repo-wide bans for `@ts-expect-error`, casts, or single-line calls as part of this workflow. + +## Extracting Proposals + +Run the extractor from the repo root: + +```sh +node scripts/extract-proposed-events.mjs --ref $(git rev-parse HEAD) +``` + +The extractor scans `packages/shared/src/telemetry/events.ts` for `@ts-expect-error` directives attached to `.track()` event-name arguments inside telemetry wrapper functions, including function declarations and variable-assigned arrow/function expressions. It emits `proposed-telemetry-extractor.v2` JSON with event names, primitive dimension names/types from the wrapper `dims` parameter type, rationale fields and missing-field flags, plus repo-relative file/line/column provenance. + +Extractor provenance is deliberately repo-relative. Absolute paths, `..` segments, Windows drive-letter paths, and backslash-separated paths are rejected so developer host paths cannot enter proposal inventory records. + +## Adoption + +When a proposed event is approved and registered in the telemetry backend, sync the regenerated telemetry artifact into the OSS repo. The event name is then part of `PaperclipEventName`, so the proposal marker should fail with TS2578. Remove the marker and keep the wrapper payload aligned with the registered dimensions in the same change. + +Old clients that do not yet have the synced schema continue to swallow the proposed event. Clients with the synced schema emit it through the normal stable telemetry path. diff --git a/packages/shared/src/telemetry/README.md b/packages/shared/src/telemetry/README.md index b81ca4eebc..ae4aad4e11 100644 --- a/packages/shared/src/telemetry/README.md +++ b/packages/shared/src/telemetry/README.md @@ -87,10 +87,16 @@ and let the receiving layer canonicalize it. ## Adding Or Changing Telemetry Client code is responsible for emitting approved telemetry events at the right -place in the product. It is not responsible for deciding which new events should -exist. Do not introduce ad hoc event names, dimensions, or enum domains in client -code; they must exist in the generated telemetry contract before emitters use -them. +place in the product. Stable event names, dimensions, and enum domains must come +from the generated telemetry contract before normal emitters use them. + +For product work that needs to propose a new first-party event before schema +registration, use the proposal marker workflow in `doc/TELEMETRY_WORKFLOW.md`. +Those proposed calls stay on `client.track()`, carry an `@ts-expect-error` +marker on the event-name argument, and are swallowed at runtime until the +generated schema registers the event name. + +For stable event work: 1. Start from `generated/paperclip-telemetry.ts`. The generated types are what reviewers use to verify event names, dimensions, optionality, value types, diff --git a/packages/shared/src/telemetry/client.test.ts b/packages/shared/src/telemetry/client.test.ts new file mode 100644 index 0000000000..87da22ba41 --- /dev/null +++ b/packages/shared/src/telemetry/client.test.ts @@ -0,0 +1,103 @@ +import { afterEach, describe, expect, it, vi } from "vitest"; +import { TelemetryClient } from "./client.js"; +import type { TelemetryConfig, TelemetryState } from "./types.js"; + +const TEST_STATE: TelemetryState = { + installId: "test-install", + salt: "test-salt", + createdAt: "2026-01-01T00:00:00Z", + firstSeenVersion: "0.0.0", +}; + +function makeClient(stateFactory = vi.fn(() => TEST_STATE), config?: Partial) { + return { + client: new TelemetryClient( + { enabled: true, endpoint: "http://localhost:9999/ingest", ...config }, + stateFactory, + "0.0.0-test", + ), + stateFactory, + }; +} + +function sentBody() { + const requestInit = vi.mocked(fetch).mock.calls.at(-1)?.[1] as RequestInit | undefined; + return JSON.parse(String(requestInit?.body ?? "{}")); +} + +describe("TelemetryClient runtime event gate", () => { + afterEach(() => { + vi.restoreAllMocks(); + }); + + it("swallows proposed first-party events before they touch state or the queue", async () => { + vi.stubGlobal("fetch", vi.fn().mockResolvedValue({ ok: true })); + const { client, stateFactory } = makeClient(); + + client.track( + // @ts-expect-error -- proposed-telemetry(PAP-2411): fixture proposal not in generated schema + "skill_studio.skill_created", + { sharing_scope: "team" }, + ); + + await client.flush(); + + expect(stateFactory).not.toHaveBeenCalled(); + expect(fetch).not.toHaveBeenCalled(); + }); + + it("uses own-property membership so prototype event names are swallowed", async () => { + vi.stubGlobal("fetch", vi.fn().mockResolvedValue({ ok: true })); + const { client, stateFactory } = makeClient(); + + // @ts-expect-error constructor is grammar-valid but not a registered Paperclip event. + client.track("constructor", {}); + // @ts-expect-error toString is grammar-valid but not a registered Paperclip event. + client.track("toString", {}); + + await client.flush(); + + expect(stateFactory).not.toHaveBeenCalled(); + expect(fetch).not.toHaveBeenCalled(); + }); + + it("keeps registered event batches unchanged", async () => { + vi.stubGlobal("fetch", vi.fn().mockResolvedValue({ ok: true })); + const { client, stateFactory } = makeClient(); + + client.track("install.started", {}); + await client.flush(); + + expect(stateFactory).toHaveBeenCalledTimes(1); + expect(fetch).toHaveBeenCalledTimes(1); + expect(sentBody()).toMatchObject({ + app: "paperclip", + schemaVersion: "1", + installId: "test-install", + version: "0.0.0-test", + events: [ + { + name: "install.started", + dimensions: {}, + }, + ], + }); + expect(sentBody().events[0]?.occurredAt).toEqual(expect.any(String)); + }); + + it("does not change trackDynamic plugin emission", async () => { + vi.stubGlobal("fetch", vi.fn().mockResolvedValue({ ok: true })); + const { client } = makeClient(); + + client.trackDynamic("plugin.linear.sync_completed", { status: "ok" }); + await client.flush(); + + expect(fetch).toHaveBeenCalledTimes(1); + expect(sentBody().events).toEqual([ + expect.objectContaining({ + name: "plugin.linear.sync_completed", + dimensions: { status: "ok" }, + }), + ]); + }); +}); diff --git a/packages/shared/src/telemetry/client.ts b/packages/shared/src/telemetry/client.ts index 178e5f0556..6af4e7f375 100644 --- a/packages/shared/src/telemetry/client.ts +++ b/packages/shared/src/telemetry/client.ts @@ -7,6 +7,7 @@ import type { TelemetryEventName, TelemetryState, } from "./types.js"; +import { PAPERCLIP_EVENTS } from "./generated/paperclip-telemetry.js"; const DEFAULT_ENDPOINTS = [ "https://telemetry.paperclip.ing/ingest", @@ -39,6 +40,7 @@ export class TelemetryClient { * backend event schema. */ track(eventName: K, ...args: TrackArgs): void { + if (!Object.hasOwn(PAPERCLIP_EVENTS, eventName)) return; const [dimensions] = args; this.enqueue(eventName, dimensions); } diff --git a/scripts/extract-proposed-events.mjs b/scripts/extract-proposed-events.mjs new file mode 100644 index 0000000000..ce29ace3fc --- /dev/null +++ b/scripts/extract-proposed-events.mjs @@ -0,0 +1,358 @@ +#!/usr/bin/env node +import { readFileSync } from "node:fs"; +import { isAbsolute, relative, resolve, sep } from "node:path"; +import { fileURLToPath } from "node:url"; +import ts from "typescript"; + +export const PROPOSED_TELEMETRY_SCHEMA_VERSION = "proposed-telemetry-extractor.v2"; + +const DEFAULT_EVENTS_FILE = "packages/shared/src/telemetry/events.ts"; +const EVENT_NAME_PATTERN = /^[a-z0-9][a-z0-9._:-]{1,63}$/; +const ISSUE_PATTERN = /^(PAP-\d+|https:\/\/github\.com\/paperclipai\/paperclip\/issues\/\d+)$/; + +export function assertRepoRelativePath(value) { + if (typeof value !== "string" || value.length === 0) { + throw new Error("provenance.file must be a non-empty repo-relative path"); + } + if (value.startsWith("/") || isAbsolute(value)) { + throw new Error(`provenance.file must be repo-relative: ${value}`); + } + if (/^[A-Za-z]:/.test(value)) { + throw new Error(`provenance.file must not use a drive-letter path: ${value}`); + } + if (value.includes("\\")) { + throw new Error(`provenance.file must use forward slashes: ${value}`); + } + const parts = value.split("/"); + if (parts.some((part) => part === "" || part === "." || part === "..")) { + throw new Error(`provenance.file contains an unsafe path segment: ${value}`); + } + if (!/^[A-Za-z0-9._-]+(\/[A-Za-z0-9._-]+)*$/.test(value)) { + throw new Error(`provenance.file contains unsupported characters: ${value}`); + } + return value; +} + +export function toRepoRelativePath(repoRoot, filePath) { + const absoluteRoot = resolve(repoRoot); + const absoluteFile = resolve(filePath); + const repoRelative = relative(absoluteRoot, absoluteFile).split(sep).join("/"); + if (repoRelative === "" || repoRelative === ".." || repoRelative.startsWith("../")) { + throw new Error(`events file must be inside repo root: ${filePath}`); + } + return assertRepoRelativePath(repoRelative); +} + +export function parseProposedTelemetryDirective(commentText) { + const normalized = normalizeComment(commentText); + if (!normalized.includes("@ts-expect-error")) return null; + + const marker = normalized.match(/@ts-expect-error\b(?:\s*--\s*)?(?:proposed-telemetry\(([^)]+)\):\s*(.*))?/s); + if (!marker) return null; + + const issue = marker[1]?.trim() || null; + const text = marker[2]?.trim() || null; + + if (issue && !ISSUE_PATTERN.test(issue)) { + throw new Error( + `proposed telemetry rationale issue must be PAP- or a paperclipai/paperclip GitHub issue URL: ${issue}`, + ); + } + + return { + issue, + text, + missingIssue: issue === null, + missingRationale: text === null, + }; +} + +export function extractProposedEvents(options = {}) { + const repoRoot = resolve(options.repoRoot ?? process.cwd()); + const eventsFile = resolve(repoRoot, options.eventsFile ?? DEFAULT_EVENTS_FILE); + const provenanceFile = toRepoRelativePath(repoRoot, eventsFile); + const sourceText = readFileSync(eventsFile, "utf8"); + const sourceFile = ts.createSourceFile(eventsFile, sourceText, ts.ScriptTarget.Latest, true, ts.ScriptKind.TS); + const proposals = new Map(); + + function recordProposal(wrapper, wrapperName, eventNameNode, directive) { + const name = eventNameNode.text; + assertEventName(name, "event name"); + const position = sourceFile.getLineAndCharacterOfPosition(eventNameNode.getStart(sourceFile)); + const dimensions = extractWrapperDimensions(wrapper, sourceFile, wrapperName); + + let proposal = proposals.get(name); + if (!proposal) { + proposal = { + name, + dimensions: new Map(), + rationale: { issue: null, text: null }, + provenance: [], + }; + proposals.set(name, proposal); + } + + mergeRationale(proposal.rationale, directive, name); + for (const dimension of dimensions) { + const existing = proposal.dimensions.get(dimension.name); + if (existing && existing.type !== dimension.type) { + throw new Error( + `conflicting inferred types for dimension ${dimension.name} on proposed event ${name}: ${existing.type} vs ${dimension.type}`, + ); + } + proposal.dimensions.set(dimension.name, dimension); + } + proposal.provenance.push({ + file: provenanceFile, + line: position.line + 1, + column: position.character, + wrapper: wrapperName, + }); + } + + function visitWrapper(wrapper, wrapperName = wrapper.name?.text ?? "") { + if (!wrapper.body) return; + + function visit(node) { + if (ts.isCallExpression(node) && isIdentifierTrackCall(node)) { + const eventNameNode = node.arguments[0]; + if (eventNameNode && ts.isStringLiteral(eventNameNode)) { + const directive = findDirectiveForNode(sourceText, eventNameNode); + if (directive) recordProposal(wrapper, wrapperName, eventNameNode, directive); + } + } + ts.forEachChild(node, visit); + } + + visit(wrapper.body); + } + + function visit(node) { + if (ts.isFunctionDeclaration(node)) { + visitWrapper(node); + } else if (ts.isVariableDeclaration(node)) { + visitVariableWrapper(node); + } + ts.forEachChild(node, visit); + } + + function visitVariableWrapper(node) { + if (!ts.isIdentifier(node.name)) return; + const initializer = node.initializer; + if (!initializer || (!ts.isArrowFunction(initializer) && !ts.isFunctionExpression(initializer))) return; + visitWrapper(initializer, node.name.text); + } + + visit(sourceFile); + + return { + schemaVersion: PROPOSED_TELEMETRY_SCHEMA_VERSION, + source: buildSource(options), + proposals: [...proposals.values()] + .map(formatProposal) + .sort((a, b) => a.name.localeCompare(b.name)), + }; +} + +function buildSource(options) { + const source = { + repo: options.repo ?? "paperclipai/paperclip", + ref: options.ref ?? process.env.GITHUB_SHA ?? process.env.PAPERCLIP_WORKSPACE_REPO_REF ?? "unknown", + }; + const baseRef = options.baseRef ?? process.env.GITHUB_BASE_REF; + if (baseRef) source.baseRef = baseRef; + return source; +} + +function isIdentifierTrackCall(node) { + const callee = node.expression; + return ( + ts.isPropertyAccessExpression(callee) && + callee.name.text === "track" && + ts.isIdentifier(callee.expression) + ); +} + +function findDirectiveForNode(sourceText, node) { + const ranges = ts.getLeadingCommentRanges(sourceText, node.getFullStart()) ?? []; + for (const range of ranges) { + const directive = parseProposedTelemetryDirective(sourceText.slice(range.pos, range.end)); + if (directive) return directive; + } + return null; +} + +function normalizeComment(commentText) { + return commentText + .replace(/^\s*\/\//, "") + .replace(/^\s*\/\*/, "") + .replace(/\*\/\s*$/, "") + .split("\n") + .map((line) => line.replace(/^\s*\*\s?/, "").trim()) + .join(" ") + .trim(); +} + +function extractWrapperDimensions(wrapper, sourceFile, wrapperName = wrapper.name?.text ?? "") { + const dimsParam = wrapper.parameters.find( + (parameter) => ts.isIdentifier(parameter.name) && parameter.name.text === "dims", + ); + if (!dimsParam) return []; + if (!dimsParam.type) { + throw new Error(`wrapper ${wrapperName} has an untyped dims parameter`); + } + const typeNode = unwrapTypeNode(dimsParam.type); + if (!ts.isTypeLiteralNode(typeNode)) { + throw new Error(`wrapper ${wrapperName} dims parameter must be a type literal`); + } + + return typeNode.members.map((member) => extractDimension(member, sourceFile)); +} + +function extractDimension(member, sourceFile) { + if (!ts.isPropertySignature(member) || !member.type) { + throw new Error("dims parameter type may contain only typed property signatures"); + } + const name = propertyNameText(member.name); + assertEventName(name, "dimension name"); + return { + name, + type: classifyTypeNode(member.type, sourceFile), + }; +} + +function propertyNameText(name) { + if (ts.isIdentifier(name) || ts.isStringLiteral(name) || ts.isNumericLiteral(name)) { + return name.text; + } + throw new Error("dimension names must be literal identifiers or string literals"); +} + +function classifyTypeNode(typeNode, sourceFile) { + const node = unwrapTypeNode(typeNode); + + if (node.kind === ts.SyntaxKind.StringKeyword) return "string"; + if (node.kind === ts.SyntaxKind.NumberKeyword) return "number"; + if (node.kind === ts.SyntaxKind.BooleanKeyword) return "boolean"; + + if (ts.isLiteralTypeNode(node)) { + const literal = node.literal; + if (ts.isStringLiteral(literal) || literal.kind === ts.SyntaxKind.NoSubstitutionTemplateLiteral) return "string"; + if (ts.isNumericLiteral(literal)) return "number"; + if (literal.kind === ts.SyntaxKind.TrueKeyword || literal.kind === ts.SyntaxKind.FalseKeyword) return "boolean"; + } + + if (ts.isTypeReferenceNode(node) && ts.isIdentifier(node.typeName) && node.typeName.text === "RawDimension") { + const inner = node.typeArguments?.[0]; + if (!inner) throw new Error(`RawDimension is missing a type argument at ${node.getText(sourceFile)}`); + return classifyTypeNode(inner, sourceFile); + } + + if (ts.isUnionTypeNode(node)) { + const primitiveTypes = new Set(); + for (const member of node.types) { + const unwrapped = unwrapTypeNode(member); + if (unwrapped.kind === ts.SyntaxKind.UndefinedKeyword) continue; + if (unwrapped.kind === ts.SyntaxKind.NullKeyword) continue; + if (ts.isLiteralTypeNode(unwrapped) && unwrapped.literal.kind === ts.SyntaxKind.NullKeyword) continue; + primitiveTypes.add(classifyTypeNode(unwrapped, sourceFile)); + } + if (primitiveTypes.size !== 1) { + throw new Error(`dimension union must resolve to one primitive type: ${node.getText(sourceFile)}`); + } + return [...primitiveTypes][0]; + } + + throw new Error(`unsupported dimension type: ${node.getText(sourceFile)}`); +} + +function unwrapTypeNode(typeNode) { + let node = typeNode; + while (ts.isParenthesizedTypeNode(node)) { + node = node.type; + } + return node; +} + +function assertEventName(value, label) { + if (!EVENT_NAME_PATTERN.test(value)) { + throw new Error(`${label} must match ${EVENT_NAME_PATTERN}: ${value}`); + } +} + +function mergeRationale(target, incoming, eventName) { + mergeRationaleField(target, incoming, "issue", eventName); + mergeRationaleField(target, incoming, "text", eventName); +} + +function mergeRationaleField(target, incoming, key, eventName) { + const value = incoming[key]; + if (!value) return; + if (target[key] && target[key] !== value) { + throw new Error(`conflicting rationale ${key} for proposed event ${eventName}`); + } + target[key] = value; +} + +function formatProposal(proposal) { + const provenance = proposal.provenance + .map((item) => ({ + file: assertRepoRelativePath(item.file), + line: item.line, + column: item.column, + })) + .sort((a, b) => a.file.localeCompare(b.file) || a.line - b.line || a.column - b.column); + + return { + name: proposal.name, + dimensions: [...proposal.dimensions.values()].sort((a, b) => a.name.localeCompare(b.name)), + rationale: { + issue: proposal.rationale.issue, + text: proposal.rationale.text, + missingIssue: proposal.rationale.issue === null, + missingRationale: proposal.rationale.text === null, + }, + provenance, + }; +} + +function parseArgs(argv) { + const options = {}; + for (let index = 0; index < argv.length; index += 1) { + const arg = argv[index]; + if (arg === "--help" || arg === "-h") { + options.help = true; + continue; + } + const next = argv[index + 1]; + if (!next || next.startsWith("--")) { + throw new Error(`${arg} requires a value`); + } + index += 1; + if (arg === "--repo-root") options.repoRoot = next; + else if (arg === "--events-file") options.eventsFile = next; + else if (arg === "--repo") options.repo = next; + else if (arg === "--ref") options.ref = next; + else if (arg === "--base-ref") options.baseRef = next; + else throw new Error(`unknown option: ${arg}`); + } + return options; +} + +function printHelp() { + process.stdout.write(`Usage: node scripts/extract-proposed-events.mjs [options]\n\nOptions:\n --repo-root Repository root. Defaults to cwd.\n --events-file events.ts path, absolute or repo-relative.\n --repo Source repository slug. Defaults to paperclipai/paperclip.\n --ref Source ref/SHA for the extractor envelope.\n --base-ref Optional base ref for diff-oriented inventory jobs.\n`); +} + +if (process.argv[1] === fileURLToPath(import.meta.url)) { + try { + const options = parseArgs(process.argv.slice(2)); + if (options.help) { + printHelp(); + process.exit(0); + } + process.stdout.write(`${JSON.stringify(extractProposedEvents(options), null, 2)}\n`); + } catch (error) { + process.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`); + process.exit(1); + } +} diff --git a/scripts/extract-proposed-events.test.mjs b/scripts/extract-proposed-events.test.mjs new file mode 100644 index 0000000000..6f3181ff9c --- /dev/null +++ b/scripts/extract-proposed-events.test.mjs @@ -0,0 +1,261 @@ +import assert from "node:assert/strict"; +import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import test from "node:test"; +import ts from "typescript"; + +import { + PROPOSED_TELEMETRY_SCHEMA_VERSION, + assertRepoRelativePath, + extractProposedEvents, + toRepoRelativePath, +} from "./extract-proposed-events.mjs"; + +function withFixtureRepo(source, callback) { + const repoRoot = mkdtempSync(join(tmpdir(), "paperclip-proposed-events-")); + const eventsFile = join(repoRoot, "packages", "shared", "src", "telemetry", "events.ts"); + mkdirSync(join(repoRoot, "packages", "shared", "src", "telemetry"), { recursive: true }); + writeFileSync(eventsFile, source); + try { + return callback({ repoRoot, eventsFile }); + } finally { + rmSync(repoRoot, { recursive: true, force: true }); + } +} + +const fixtureSource = ` +import type { TelemetryClient } from "./client.js"; + +type RawDimension = T | (string & {}); + +export function trackSkillStudioCreated( + client: TelemetryClient, + dims: { + sharing_scope: RawDimension<"team" | "private">; + category_count: number; + launched_from_template: boolean; + }, +): void { + client.track( + // @ts-expect-error -- proposed-telemetry(PAP-2411): measure Skill Studio create completion + "skill_studio.skill_created", + dims, + ); +} + +export function trackSkillStudioOpened( + client: TelemetryClient, + dims: { + surface: "modal" | "page"; + }, +): void { + client.track( + // @ts-expect-error + "skill_studio.opened", + dims, + ); +} + +export function trackInstallStarted(client: TelemetryClient): void { + client.track("install.started", {}); +} +`; + +test("extractor emits deterministic proposed-telemetry-extractor.v2 records", () => { + const output = withFixtureRepo(fixtureSource, ({ repoRoot, eventsFile }) => + extractProposedEvents({ repoRoot, eventsFile, ref: "fixture-sha", baseRef: "master" }), + ); + + assert.equal(output.schemaVersion, PROPOSED_TELEMETRY_SCHEMA_VERSION); + assert.deepEqual(output.source, { + repo: "paperclipai/paperclip", + ref: "fixture-sha", + baseRef: "master", + }); + assert.deepEqual( + output.proposals.map((proposal) => proposal.name), + ["skill_studio.opened", "skill_studio.skill_created"], + ); + + const created = output.proposals.find((proposal) => proposal.name === "skill_studio.skill_created"); + assert.deepEqual(created.dimensions, [ + { name: "category_count", type: "number" }, + { name: "launched_from_template", type: "boolean" }, + { name: "sharing_scope", type: "string" }, + ]); + assert.deepEqual(created.rationale, { + issue: "PAP-2411", + text: "measure Skill Studio create completion", + missingIssue: false, + missingRationale: false, + }); + assert.equal(created.provenance.length, 1); + assert.equal(created.provenance[0].file, "packages/shared/src/telemetry/events.ts"); + assert.equal(typeof created.provenance[0].line, "number"); + assert.equal(typeof created.provenance[0].column, "number"); +}); + +test("extractor flags a missing proposed-telemetry suffix without hard-failing", () => { + const output = withFixtureRepo(fixtureSource, ({ repoRoot, eventsFile }) => + extractProposedEvents({ repoRoot, eventsFile, ref: "fixture-sha" }), + ); + + const opened = output.proposals.find((proposal) => proposal.name === "skill_studio.opened"); + assert.deepEqual(opened.rationale, { + issue: null, + text: null, + missingIssue: true, + missingRationale: true, + }); + assert.deepEqual(opened.dimensions, [{ name: "surface", type: "string" }]); +}); + +test("extractor scans TelemetryClient wrappers whose receiver is not named client", () => { + const output = withFixtureRepo( + `type TelemetryClient = { track(name: string, dims: unknown): void }; + +export function trackWorkspaceOpened( + telemetry: TelemetryClient, + dims: { surface: string }, +): void { + telemetry.track( + // @ts-expect-error -- proposed-telemetry(PAP-2463): exercise alternate telemetry client parameter names + "workspace.opened", + dims, + ); +} +`, + ({ repoRoot, eventsFile }) => extractProposedEvents({ repoRoot, eventsFile, ref: "fixture-sha" }), + ); + + assert.deepEqual( + output.proposals.map((proposal) => proposal.name), + ["workspace.opened"], + ); + assert.deepEqual(output.proposals[0].dimensions, [{ name: "surface", type: "string" }]); +}); + +test("extractor scans variable-assigned wrappers and ignores nullable union members", () => { + const output = withFixtureRepo( + `type TelemetryClient = { track(name: string, dims: unknown): void }; + +export const trackWorkspaceArrow = ( + telemetry: TelemetryClient, + dims: { surface: string | null | undefined }, +): void => { + telemetry.track( + // @ts-expect-error -- proposed-telemetry(PAP-2463): exercise arrow wrapper extraction + "workspace.arrow_opened", + dims, + ); +}; + +export const trackWorkspaceFunctionExpression = function ( + tc: TelemetryClient, + dims: { accepted: true | false | null }, +): void { + tc.track( + // @ts-expect-error -- proposed-telemetry(PAP-2463): exercise function-expression wrapper extraction + "workspace.function_expression_opened", + dims, + ); +}; +`, + ({ repoRoot, eventsFile }) => extractProposedEvents({ repoRoot, eventsFile, ref: "fixture-sha" }), + ); + + assert.deepEqual( + output.proposals.map((proposal) => proposal.name), + ["workspace.arrow_opened", "workspace.function_expression_opened"], + ); + assert.deepEqual(output.proposals[0].dimensions, [{ name: "surface", type: "string" }]); + assert.deepEqual(output.proposals[1].dimensions, [{ name: "accepted", type: "boolean" }]); +}); + +test("extractor rejects invalid rationale issue references when present", () => { + assert.throws( + () => + withFixtureRepo( + `export function trackBad(client, dims: { source: string }): void {\n client.track(\n // @ts-expect-error -- proposed-telemetry(PROJ-1): bad issue ref\n "skill_studio.bad_issue",\n dims,\n );\n}\n`, + ({ repoRoot, eventsFile }) => extractProposedEvents({ repoRoot, eventsFile, ref: "fixture-sha" }), + ), + /rationale issue must be PAP-/, + ); +}); + +test("provenance paths are repo-relative and reject dev-host path shapes", () => { + assert.equal( + assertRepoRelativePath("packages/shared/src/telemetry/events.ts"), + "packages/shared/src/telemetry/events.ts", + ); + assert.throws(() => assertRepoRelativePath("/tmp/events.ts"), /repo-relative/); + assert.throws(() => assertRepoRelativePath("../events.ts"), /unsafe path segment/); + assert.throws(() => assertRepoRelativePath("C:/repo/events.ts"), /drive-letter/); + assert.throws(() => assertRepoRelativePath("packages\\shared\\events.ts"), /forward slashes/); + + const repoRoot = mkdtempSync(join(tmpdir(), "paperclip-provenance-root-")); + try { + assert.throws(() => toRepoRelativePath(repoRoot, join(repoRoot, "..", "events.ts")), /inside repo root/); + } finally { + rmSync(repoRoot, { recursive: true, force: true }); + } +}); + +function diagnosticsFor(sourceText) { + const repoRoot = mkdtempSync(join(tmpdir(), "paperclip-ts2578-")); + const fileName = join(repoRoot, "fixture.ts"); + writeFileSync(fileName, sourceText); + try { + const program = ts.createProgram([fileName], { + strict: true, + noEmit: true, + skipLibCheck: true, + target: ts.ScriptTarget.ES2023, + module: ts.ModuleKind.NodeNext, + moduleResolution: ts.ModuleResolutionKind.NodeNext, + types: [], + }); + return ts + .getPreEmitDiagnostics(program) + .filter((diagnostic) => diagnostic.file?.fileName === fileName) + .map((diagnostic) => diagnostic.code); + } finally { + rmSync(repoRoot, { recursive: true, force: true }); + } +} + +function tsMechanicsFixture(eventUnion, mapEntry) { + return ` +type TelemetryEventName = ${eventUnion}; +interface EventDimensionsMap { + "install.started": {}; + ${mapEntry} +} +type TelemetryEventDimensions = EventDimensionsMap[K]; +type TrackArgs = keyof TelemetryEventDimensions extends never + ? [dimensions?: TelemetryEventDimensions] + : [dimensions: TelemetryEventDimensions]; +declare const client: { + track(eventName: K, ...args: TrackArgs): void; +}; +client.track( + // @ts-expect-error -- proposed-telemetry(PAP-2411): TS2578 expiry fixture + "skill_studio.skill_created", + { sharing_scope: "team" }, +); +`; +} + +test("TS2578 expires the directive once a fixture event is registered", () => { + const unregistered = diagnosticsFor(tsMechanicsFixture('"install.started"', "")); + assert.deepEqual(unregistered, []); + + const registered = diagnosticsFor( + tsMechanicsFixture( + '"install.started" | "skill_studio.skill_created"', + '"skill_studio.skill_created": { sharing_scope: string };', + ), + ); + assert.ok(registered.includes(2578), `expected TS2578, got ${registered.join(", ")}`); +});