Add telemetry proposal extractor (#9544)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - It emits telemetry events to understand product usage — registered event names are gated by a generated `PAPERCLIP_EVENTS` registry so the client only enqueues known, schema-approved events > - When product teams want to instrument a new behaviour, they must first register the event name — but schema registration is a commit-and-release cycle, which creates friction in fast-moving product iterations > - A proposal lane is needed: let developers mark a `track()` call with a typed `@ts-expect-error` proposal marker so the event name can be reviewed and tracked in CI before the schema is formally registered > - The existing client had no guard against unregistered event names, so any call with an out-of-registry name (or a prototype-inherited key) would silently enter the queue, state, and network flush path > - This PR adds an `Object.hasOwn(PAPERCLIP_EVENTS, eventName)` guard at the entry point of `track()` to swallow unregistered calls before any side effects, adds `scripts/extract-proposed-events.mjs` to scan source for proposal markers and emit a v2 JSON manifest with provenance and rationale, and documents the complete proposal workflow > - The benefit is that new instrumentation can be proposed and reviewed in code without touching the registered schema, and tooling can surface missing rationale before events graduate to stable ## Linked Issues or Issue Description No existing GitHub issue covers this change. This PR introduces a new feature. **Feature motivation:** Paperclip's telemetry schema is intentionally stable — registered event names are code-generated and gated. Product engineers who want to instrument a new behaviour today must land a schema change first, creating a two-step process that slows iteration. A proposal lane lets developers write the instrumentation call ahead of schema registration, protected by a compile-time `@ts-expect-error` marker that an extractor script can surface for review. This PR implements both the client-side safety gate and the extraction tooling. Refs: #9518 (closed predecessor — docs-only; this PR supersedes it with the full implementation) ## What Changed - Added `Object.hasOwn(PAPERCLIP_EVENTS, eventName)` guard at the top of `TelemetryClient.track()`: unregistered event names (including prototype-inherited keys) are now swallowed before any state, queue, or network operation - Added `scripts/extract-proposed-events.mjs`: scans TypeScript source for `@ts-expect-error -- proposed-telemetry(<issue>): <rationale>` markers; emits a v2 JSON manifest per proposed event including name, rationale, provenance (repo-relative file + line), and a `rationale_missing` flag for CI enforcement - Added `scripts/extract-proposed-events.test.mjs`: test suite covering marker parsing, multi-line markers, path validation, out-of-repo rejection, and the v2 schema output contract - Added `doc/TELEMETRY_WORKFLOW.md`: documents the proposal workflow, the canonical multi-line marker example, rationale requirements, and how to graduate a proposed event to stable schema - Updated `packages/shared/src/telemetry/README.md`: added "Proposed Events" section to the Telemetry Data Contract per the contributing guide requirement for telemetry changes ## Verification Run all of the following from the repo root: ```sh # Extractor unit tests node --test scripts/extract-proposed-events.test.mjs # Telemetry client + types tests pnpm exec vitest run --config vitest.config.ts \ src/telemetry/client.test.ts src/telemetry/client-types.test.ts \ --reporter=verbose # (run from packages/shared) # Type-check pnpm --filter @paperclipai/shared typecheck # Smoke-run the extractor in local-test mode node scripts/extract-proposed-events.mjs --ref local-test ``` All four commands pass locally. ## Risks - **Silent drop on unregistered events:** The `Object.hasOwn` guard fails closed — any event name not in `PAPERCLIP_EVENTS` is silently dropped. If the generated registry is missing an event that was previously tracked, those calls will be silently lost. Mitigation: the extractor script surfaces proposed events that need registration; the TypeScript type system already enforces `TelemetryEventName ⊆ PAPERCLIP_EVENTS` at compile time. - **Extractor is read-only:** `extract-proposed-events.mjs` reads source and emits JSON; it does not modify any files. No runtime or schema risk. - Overall risk: **low**. The guard is additive and defensive; the extractor and docs are additive only. ## Model Used - Provider: Anthropic - Model ID: `claude-sonnet-4-6` - Context window: 200 K tokens - Capabilities: tool use, extended context, code generation ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Nicky Leach
GitHub
parent
6a36ae47fe
commit
90f85a7d11
|
|
@ -0,0 +1,59 @@
|
|||
# Telemetry Workflow
|
||||
|
||||
Paperclip first-party telemetry is schema-led for stable events and proposal-led for new product instrumentation.
|
||||
|
||||
Stable events must be present in `packages/shared/src/telemetry/generated/paperclip-telemetry.ts` before normal client code emits them. Proposed events may be added ahead of schema registration only with an `@ts-expect-error` proposal marker on the `client.track()` event-name argument.
|
||||
|
||||
## Proposed Events
|
||||
|
||||
A proposed event is a normal `client.track()` call whose event name is not yet in `PaperclipEventName`. The runtime client swallows unregistered first-party event names before queueing, state initialization, or network flush, so proposed events do not leave the process until the generated telemetry schema adopts the event name.
|
||||
|
||||
Use this marker shape when possible:
|
||||
|
||||
```ts
|
||||
import type { TelemetryClient } from "./client.js";
|
||||
|
||||
export function trackYourFeatureActionPerformed(
|
||||
client: TelemetryClient,
|
||||
dims: {
|
||||
action_source: "toolbar" | "menu" | (string & {});
|
||||
item_count: number;
|
||||
},
|
||||
): void {
|
||||
client.track(
|
||||
// @ts-expect-error -- proposed-telemetry(https://github.com/paperclipai/paperclip/issues/123): measure feature action completion
|
||||
"your_feature.action_performed",
|
||||
dims,
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
The multi-line shape is recommended because TypeScript places the unregistered-name error on the event-name line. When the schema later registers the event, that error disappears and TypeScript raises TS2578 for the now-unused directive, which tells the adopter to remove the marker in the same change that syncs the generated schema.
|
||||
|
||||
The suffix format is:
|
||||
|
||||
```text
|
||||
-- proposed-telemetry(<issue>): <rationale>
|
||||
```
|
||||
|
||||
`<issue>` should be a public `https://github.com/paperclipai/paperclip/issues/123` URL. The rationale should be a short product reason for collecting the event. Missing issue or rationale text is tolerated at the call site and flagged by `scripts/extract-proposed-events.mjs`; it is not an OSS CI failure.
|
||||
|
||||
These formatting conventions are documentation-only. Do not add repo-wide bans for `@ts-expect-error`, casts, or single-line calls as part of this workflow.
|
||||
|
||||
## Extracting Proposals
|
||||
|
||||
Run the extractor from the repo root:
|
||||
|
||||
```sh
|
||||
node scripts/extract-proposed-events.mjs --ref $(git rev-parse HEAD)
|
||||
```
|
||||
|
||||
The extractor scans `packages/shared/src/telemetry/events.ts` for `@ts-expect-error` directives attached to `<identifier>.track()` event-name arguments inside telemetry wrapper functions, including function declarations and variable-assigned arrow/function expressions. It emits `proposed-telemetry-extractor.v2` JSON with event names, primitive dimension names/types from the wrapper `dims` parameter type, rationale fields and missing-field flags, plus repo-relative file/line/column provenance.
|
||||
|
||||
Extractor provenance is deliberately repo-relative. Absolute paths, `..` segments, Windows drive-letter paths, and backslash-separated paths are rejected so developer host paths cannot enter proposal inventory records.
|
||||
|
||||
## Adoption
|
||||
|
||||
When a proposed event is approved and registered in the telemetry backend, sync the regenerated telemetry artifact into the OSS repo. The event name is then part of `PaperclipEventName`, so the proposal marker should fail with TS2578. Remove the marker and keep the wrapper payload aligned with the registered dimensions in the same change.
|
||||
|
||||
Old clients that do not yet have the synced schema continue to swallow the proposed event. Clients with the synced schema emit it through the normal stable telemetry path.
|
||||
|
|
@ -87,10 +87,16 @@ and let the receiving layer canonicalize it.
|
|||
## Adding Or Changing Telemetry
|
||||
|
||||
Client code is responsible for emitting approved telemetry events at the right
|
||||
place in the product. It is not responsible for deciding which new events should
|
||||
exist. Do not introduce ad hoc event names, dimensions, or enum domains in client
|
||||
code; they must exist in the generated telemetry contract before emitters use
|
||||
them.
|
||||
place in the product. Stable event names, dimensions, and enum domains must come
|
||||
from the generated telemetry contract before normal emitters use them.
|
||||
|
||||
For product work that needs to propose a new first-party event before schema
|
||||
registration, use the proposal marker workflow in `doc/TELEMETRY_WORKFLOW.md`.
|
||||
Those proposed calls stay on `client.track()`, carry an `@ts-expect-error`
|
||||
marker on the event-name argument, and are swallowed at runtime until the
|
||||
generated schema registers the event name.
|
||||
|
||||
For stable event work:
|
||||
|
||||
1. Start from `generated/paperclip-telemetry.ts`. The generated types are what
|
||||
reviewers use to verify event names, dimensions, optionality, value types,
|
||||
|
|
|
|||
|
|
@ -0,0 +1,103 @@
|
|||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { TelemetryClient } from "./client.js";
|
||||
import type { TelemetryConfig, TelemetryState } from "./types.js";
|
||||
|
||||
const TEST_STATE: TelemetryState = {
|
||||
installId: "test-install",
|
||||
salt: "test-salt",
|
||||
createdAt: "2026-01-01T00:00:00Z",
|
||||
firstSeenVersion: "0.0.0",
|
||||
};
|
||||
|
||||
function makeClient(stateFactory = vi.fn(() => TEST_STATE), config?: Partial<TelemetryConfig>) {
|
||||
return {
|
||||
client: new TelemetryClient(
|
||||
{ enabled: true, endpoint: "http://localhost:9999/ingest", ...config },
|
||||
stateFactory,
|
||||
"0.0.0-test",
|
||||
),
|
||||
stateFactory,
|
||||
};
|
||||
}
|
||||
|
||||
function sentBody() {
|
||||
const requestInit = vi.mocked(fetch).mock.calls.at(-1)?.[1] as RequestInit | undefined;
|
||||
return JSON.parse(String(requestInit?.body ?? "{}"));
|
||||
}
|
||||
|
||||
describe("TelemetryClient runtime event gate", () => {
|
||||
afterEach(() => {
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
it("swallows proposed first-party events before they touch state or the queue", async () => {
|
||||
vi.stubGlobal("fetch", vi.fn().mockResolvedValue({ ok: true }));
|
||||
const { client, stateFactory } = makeClient();
|
||||
|
||||
client.track(
|
||||
// @ts-expect-error -- proposed-telemetry(PAP-2411): fixture proposal not in generated schema
|
||||
"skill_studio.skill_created",
|
||||
{ sharing_scope: "team" },
|
||||
);
|
||||
|
||||
await client.flush();
|
||||
|
||||
expect(stateFactory).not.toHaveBeenCalled();
|
||||
expect(fetch).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("uses own-property membership so prototype event names are swallowed", async () => {
|
||||
vi.stubGlobal("fetch", vi.fn().mockResolvedValue({ ok: true }));
|
||||
const { client, stateFactory } = makeClient();
|
||||
|
||||
// @ts-expect-error constructor is grammar-valid but not a registered Paperclip event.
|
||||
client.track("constructor", {});
|
||||
// @ts-expect-error toString is grammar-valid but not a registered Paperclip event.
|
||||
client.track("toString", {});
|
||||
|
||||
await client.flush();
|
||||
|
||||
expect(stateFactory).not.toHaveBeenCalled();
|
||||
expect(fetch).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("keeps registered event batches unchanged", async () => {
|
||||
vi.stubGlobal("fetch", vi.fn().mockResolvedValue({ ok: true }));
|
||||
const { client, stateFactory } = makeClient();
|
||||
|
||||
client.track("install.started", {});
|
||||
await client.flush();
|
||||
|
||||
expect(stateFactory).toHaveBeenCalledTimes(1);
|
||||
expect(fetch).toHaveBeenCalledTimes(1);
|
||||
expect(sentBody()).toMatchObject({
|
||||
app: "paperclip",
|
||||
schemaVersion: "1",
|
||||
installId: "test-install",
|
||||
version: "0.0.0-test",
|
||||
events: [
|
||||
{
|
||||
name: "install.started",
|
||||
dimensions: {},
|
||||
},
|
||||
],
|
||||
});
|
||||
expect(sentBody().events[0]?.occurredAt).toEqual(expect.any(String));
|
||||
});
|
||||
|
||||
it("does not change trackDynamic plugin emission", async () => {
|
||||
vi.stubGlobal("fetch", vi.fn().mockResolvedValue({ ok: true }));
|
||||
const { client } = makeClient();
|
||||
|
||||
client.trackDynamic("plugin.linear.sync_completed", { status: "ok" });
|
||||
await client.flush();
|
||||
|
||||
expect(fetch).toHaveBeenCalledTimes(1);
|
||||
expect(sentBody().events).toEqual([
|
||||
expect.objectContaining({
|
||||
name: "plugin.linear.sync_completed",
|
||||
dimensions: { status: "ok" },
|
||||
}),
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
|
@ -7,6 +7,7 @@ import type {
|
|||
TelemetryEventName,
|
||||
TelemetryState,
|
||||
} from "./types.js";
|
||||
import { PAPERCLIP_EVENTS } from "./generated/paperclip-telemetry.js";
|
||||
|
||||
const DEFAULT_ENDPOINTS = [
|
||||
"https://telemetry.paperclip.ing/ingest",
|
||||
|
|
@ -39,6 +40,7 @@ export class TelemetryClient {
|
|||
* backend event schema.
|
||||
*/
|
||||
track<K extends TelemetryEventName>(eventName: K, ...args: TrackArgs<K>): void {
|
||||
if (!Object.hasOwn(PAPERCLIP_EVENTS, eventName)) return;
|
||||
const [dimensions] = args;
|
||||
this.enqueue(eventName, dimensions);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,358 @@
|
|||
#!/usr/bin/env node
|
||||
import { readFileSync } from "node:fs";
|
||||
import { isAbsolute, relative, resolve, sep } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import ts from "typescript";
|
||||
|
||||
export const PROPOSED_TELEMETRY_SCHEMA_VERSION = "proposed-telemetry-extractor.v2";
|
||||
|
||||
const DEFAULT_EVENTS_FILE = "packages/shared/src/telemetry/events.ts";
|
||||
const EVENT_NAME_PATTERN = /^[a-z0-9][a-z0-9._:-]{1,63}$/;
|
||||
const ISSUE_PATTERN = /^(PAP-\d+|https:\/\/github\.com\/paperclipai\/paperclip\/issues\/\d+)$/;
|
||||
|
||||
export function assertRepoRelativePath(value) {
|
||||
if (typeof value !== "string" || value.length === 0) {
|
||||
throw new Error("provenance.file must be a non-empty repo-relative path");
|
||||
}
|
||||
if (value.startsWith("/") || isAbsolute(value)) {
|
||||
throw new Error(`provenance.file must be repo-relative: ${value}`);
|
||||
}
|
||||
if (/^[A-Za-z]:/.test(value)) {
|
||||
throw new Error(`provenance.file must not use a drive-letter path: ${value}`);
|
||||
}
|
||||
if (value.includes("\\")) {
|
||||
throw new Error(`provenance.file must use forward slashes: ${value}`);
|
||||
}
|
||||
const parts = value.split("/");
|
||||
if (parts.some((part) => part === "" || part === "." || part === "..")) {
|
||||
throw new Error(`provenance.file contains an unsafe path segment: ${value}`);
|
||||
}
|
||||
if (!/^[A-Za-z0-9._-]+(\/[A-Za-z0-9._-]+)*$/.test(value)) {
|
||||
throw new Error(`provenance.file contains unsupported characters: ${value}`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
export function toRepoRelativePath(repoRoot, filePath) {
|
||||
const absoluteRoot = resolve(repoRoot);
|
||||
const absoluteFile = resolve(filePath);
|
||||
const repoRelative = relative(absoluteRoot, absoluteFile).split(sep).join("/");
|
||||
if (repoRelative === "" || repoRelative === ".." || repoRelative.startsWith("../")) {
|
||||
throw new Error(`events file must be inside repo root: ${filePath}`);
|
||||
}
|
||||
return assertRepoRelativePath(repoRelative);
|
||||
}
|
||||
|
||||
export function parseProposedTelemetryDirective(commentText) {
|
||||
const normalized = normalizeComment(commentText);
|
||||
if (!normalized.includes("@ts-expect-error")) return null;
|
||||
|
||||
const marker = normalized.match(/@ts-expect-error\b(?:\s*--\s*)?(?:proposed-telemetry\(([^)]+)\):\s*(.*))?/s);
|
||||
if (!marker) return null;
|
||||
|
||||
const issue = marker[1]?.trim() || null;
|
||||
const text = marker[2]?.trim() || null;
|
||||
|
||||
if (issue && !ISSUE_PATTERN.test(issue)) {
|
||||
throw new Error(
|
||||
`proposed telemetry rationale issue must be PAP-<digits> or a paperclipai/paperclip GitHub issue URL: ${issue}`,
|
||||
);
|
||||
}
|
||||
|
||||
return {
|
||||
issue,
|
||||
text,
|
||||
missingIssue: issue === null,
|
||||
missingRationale: text === null,
|
||||
};
|
||||
}
|
||||
|
||||
export function extractProposedEvents(options = {}) {
|
||||
const repoRoot = resolve(options.repoRoot ?? process.cwd());
|
||||
const eventsFile = resolve(repoRoot, options.eventsFile ?? DEFAULT_EVENTS_FILE);
|
||||
const provenanceFile = toRepoRelativePath(repoRoot, eventsFile);
|
||||
const sourceText = readFileSync(eventsFile, "utf8");
|
||||
const sourceFile = ts.createSourceFile(eventsFile, sourceText, ts.ScriptTarget.Latest, true, ts.ScriptKind.TS);
|
||||
const proposals = new Map();
|
||||
|
||||
function recordProposal(wrapper, wrapperName, eventNameNode, directive) {
|
||||
const name = eventNameNode.text;
|
||||
assertEventName(name, "event name");
|
||||
const position = sourceFile.getLineAndCharacterOfPosition(eventNameNode.getStart(sourceFile));
|
||||
const dimensions = extractWrapperDimensions(wrapper, sourceFile, wrapperName);
|
||||
|
||||
let proposal = proposals.get(name);
|
||||
if (!proposal) {
|
||||
proposal = {
|
||||
name,
|
||||
dimensions: new Map(),
|
||||
rationale: { issue: null, text: null },
|
||||
provenance: [],
|
||||
};
|
||||
proposals.set(name, proposal);
|
||||
}
|
||||
|
||||
mergeRationale(proposal.rationale, directive, name);
|
||||
for (const dimension of dimensions) {
|
||||
const existing = proposal.dimensions.get(dimension.name);
|
||||
if (existing && existing.type !== dimension.type) {
|
||||
throw new Error(
|
||||
`conflicting inferred types for dimension ${dimension.name} on proposed event ${name}: ${existing.type} vs ${dimension.type}`,
|
||||
);
|
||||
}
|
||||
proposal.dimensions.set(dimension.name, dimension);
|
||||
}
|
||||
proposal.provenance.push({
|
||||
file: provenanceFile,
|
||||
line: position.line + 1,
|
||||
column: position.character,
|
||||
wrapper: wrapperName,
|
||||
});
|
||||
}
|
||||
|
||||
function visitWrapper(wrapper, wrapperName = wrapper.name?.text ?? "<anonymous>") {
|
||||
if (!wrapper.body) return;
|
||||
|
||||
function visit(node) {
|
||||
if (ts.isCallExpression(node) && isIdentifierTrackCall(node)) {
|
||||
const eventNameNode = node.arguments[0];
|
||||
if (eventNameNode && ts.isStringLiteral(eventNameNode)) {
|
||||
const directive = findDirectiveForNode(sourceText, eventNameNode);
|
||||
if (directive) recordProposal(wrapper, wrapperName, eventNameNode, directive);
|
||||
}
|
||||
}
|
||||
ts.forEachChild(node, visit);
|
||||
}
|
||||
|
||||
visit(wrapper.body);
|
||||
}
|
||||
|
||||
function visit(node) {
|
||||
if (ts.isFunctionDeclaration(node)) {
|
||||
visitWrapper(node);
|
||||
} else if (ts.isVariableDeclaration(node)) {
|
||||
visitVariableWrapper(node);
|
||||
}
|
||||
ts.forEachChild(node, visit);
|
||||
}
|
||||
|
||||
function visitVariableWrapper(node) {
|
||||
if (!ts.isIdentifier(node.name)) return;
|
||||
const initializer = node.initializer;
|
||||
if (!initializer || (!ts.isArrowFunction(initializer) && !ts.isFunctionExpression(initializer))) return;
|
||||
visitWrapper(initializer, node.name.text);
|
||||
}
|
||||
|
||||
visit(sourceFile);
|
||||
|
||||
return {
|
||||
schemaVersion: PROPOSED_TELEMETRY_SCHEMA_VERSION,
|
||||
source: buildSource(options),
|
||||
proposals: [...proposals.values()]
|
||||
.map(formatProposal)
|
||||
.sort((a, b) => a.name.localeCompare(b.name)),
|
||||
};
|
||||
}
|
||||
|
||||
function buildSource(options) {
|
||||
const source = {
|
||||
repo: options.repo ?? "paperclipai/paperclip",
|
||||
ref: options.ref ?? process.env.GITHUB_SHA ?? process.env.PAPERCLIP_WORKSPACE_REPO_REF ?? "unknown",
|
||||
};
|
||||
const baseRef = options.baseRef ?? process.env.GITHUB_BASE_REF;
|
||||
if (baseRef) source.baseRef = baseRef;
|
||||
return source;
|
||||
}
|
||||
|
||||
function isIdentifierTrackCall(node) {
|
||||
const callee = node.expression;
|
||||
return (
|
||||
ts.isPropertyAccessExpression(callee) &&
|
||||
callee.name.text === "track" &&
|
||||
ts.isIdentifier(callee.expression)
|
||||
);
|
||||
}
|
||||
|
||||
function findDirectiveForNode(sourceText, node) {
|
||||
const ranges = ts.getLeadingCommentRanges(sourceText, node.getFullStart()) ?? [];
|
||||
for (const range of ranges) {
|
||||
const directive = parseProposedTelemetryDirective(sourceText.slice(range.pos, range.end));
|
||||
if (directive) return directive;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function normalizeComment(commentText) {
|
||||
return commentText
|
||||
.replace(/^\s*\/\//, "")
|
||||
.replace(/^\s*\/\*/, "")
|
||||
.replace(/\*\/\s*$/, "")
|
||||
.split("\n")
|
||||
.map((line) => line.replace(/^\s*\*\s?/, "").trim())
|
||||
.join(" ")
|
||||
.trim();
|
||||
}
|
||||
|
||||
function extractWrapperDimensions(wrapper, sourceFile, wrapperName = wrapper.name?.text ?? "<anonymous>") {
|
||||
const dimsParam = wrapper.parameters.find(
|
||||
(parameter) => ts.isIdentifier(parameter.name) && parameter.name.text === "dims",
|
||||
);
|
||||
if (!dimsParam) return [];
|
||||
if (!dimsParam.type) {
|
||||
throw new Error(`wrapper ${wrapperName} has an untyped dims parameter`);
|
||||
}
|
||||
const typeNode = unwrapTypeNode(dimsParam.type);
|
||||
if (!ts.isTypeLiteralNode(typeNode)) {
|
||||
throw new Error(`wrapper ${wrapperName} dims parameter must be a type literal`);
|
||||
}
|
||||
|
||||
return typeNode.members.map((member) => extractDimension(member, sourceFile));
|
||||
}
|
||||
|
||||
function extractDimension(member, sourceFile) {
|
||||
if (!ts.isPropertySignature(member) || !member.type) {
|
||||
throw new Error("dims parameter type may contain only typed property signatures");
|
||||
}
|
||||
const name = propertyNameText(member.name);
|
||||
assertEventName(name, "dimension name");
|
||||
return {
|
||||
name,
|
||||
type: classifyTypeNode(member.type, sourceFile),
|
||||
};
|
||||
}
|
||||
|
||||
function propertyNameText(name) {
|
||||
if (ts.isIdentifier(name) || ts.isStringLiteral(name) || ts.isNumericLiteral(name)) {
|
||||
return name.text;
|
||||
}
|
||||
throw new Error("dimension names must be literal identifiers or string literals");
|
||||
}
|
||||
|
||||
function classifyTypeNode(typeNode, sourceFile) {
|
||||
const node = unwrapTypeNode(typeNode);
|
||||
|
||||
if (node.kind === ts.SyntaxKind.StringKeyword) return "string";
|
||||
if (node.kind === ts.SyntaxKind.NumberKeyword) return "number";
|
||||
if (node.kind === ts.SyntaxKind.BooleanKeyword) return "boolean";
|
||||
|
||||
if (ts.isLiteralTypeNode(node)) {
|
||||
const literal = node.literal;
|
||||
if (ts.isStringLiteral(literal) || literal.kind === ts.SyntaxKind.NoSubstitutionTemplateLiteral) return "string";
|
||||
if (ts.isNumericLiteral(literal)) return "number";
|
||||
if (literal.kind === ts.SyntaxKind.TrueKeyword || literal.kind === ts.SyntaxKind.FalseKeyword) return "boolean";
|
||||
}
|
||||
|
||||
if (ts.isTypeReferenceNode(node) && ts.isIdentifier(node.typeName) && node.typeName.text === "RawDimension") {
|
||||
const inner = node.typeArguments?.[0];
|
||||
if (!inner) throw new Error(`RawDimension is missing a type argument at ${node.getText(sourceFile)}`);
|
||||
return classifyTypeNode(inner, sourceFile);
|
||||
}
|
||||
|
||||
if (ts.isUnionTypeNode(node)) {
|
||||
const primitiveTypes = new Set();
|
||||
for (const member of node.types) {
|
||||
const unwrapped = unwrapTypeNode(member);
|
||||
if (unwrapped.kind === ts.SyntaxKind.UndefinedKeyword) continue;
|
||||
if (unwrapped.kind === ts.SyntaxKind.NullKeyword) continue;
|
||||
if (ts.isLiteralTypeNode(unwrapped) && unwrapped.literal.kind === ts.SyntaxKind.NullKeyword) continue;
|
||||
primitiveTypes.add(classifyTypeNode(unwrapped, sourceFile));
|
||||
}
|
||||
if (primitiveTypes.size !== 1) {
|
||||
throw new Error(`dimension union must resolve to one primitive type: ${node.getText(sourceFile)}`);
|
||||
}
|
||||
return [...primitiveTypes][0];
|
||||
}
|
||||
|
||||
throw new Error(`unsupported dimension type: ${node.getText(sourceFile)}`);
|
||||
}
|
||||
|
||||
function unwrapTypeNode(typeNode) {
|
||||
let node = typeNode;
|
||||
while (ts.isParenthesizedTypeNode(node)) {
|
||||
node = node.type;
|
||||
}
|
||||
return node;
|
||||
}
|
||||
|
||||
function assertEventName(value, label) {
|
||||
if (!EVENT_NAME_PATTERN.test(value)) {
|
||||
throw new Error(`${label} must match ${EVENT_NAME_PATTERN}: ${value}`);
|
||||
}
|
||||
}
|
||||
|
||||
function mergeRationale(target, incoming, eventName) {
|
||||
mergeRationaleField(target, incoming, "issue", eventName);
|
||||
mergeRationaleField(target, incoming, "text", eventName);
|
||||
}
|
||||
|
||||
function mergeRationaleField(target, incoming, key, eventName) {
|
||||
const value = incoming[key];
|
||||
if (!value) return;
|
||||
if (target[key] && target[key] !== value) {
|
||||
throw new Error(`conflicting rationale ${key} for proposed event ${eventName}`);
|
||||
}
|
||||
target[key] = value;
|
||||
}
|
||||
|
||||
function formatProposal(proposal) {
|
||||
const provenance = proposal.provenance
|
||||
.map((item) => ({
|
||||
file: assertRepoRelativePath(item.file),
|
||||
line: item.line,
|
||||
column: item.column,
|
||||
}))
|
||||
.sort((a, b) => a.file.localeCompare(b.file) || a.line - b.line || a.column - b.column);
|
||||
|
||||
return {
|
||||
name: proposal.name,
|
||||
dimensions: [...proposal.dimensions.values()].sort((a, b) => a.name.localeCompare(b.name)),
|
||||
rationale: {
|
||||
issue: proposal.rationale.issue,
|
||||
text: proposal.rationale.text,
|
||||
missingIssue: proposal.rationale.issue === null,
|
||||
missingRationale: proposal.rationale.text === null,
|
||||
},
|
||||
provenance,
|
||||
};
|
||||
}
|
||||
|
||||
function parseArgs(argv) {
|
||||
const options = {};
|
||||
for (let index = 0; index < argv.length; index += 1) {
|
||||
const arg = argv[index];
|
||||
if (arg === "--help" || arg === "-h") {
|
||||
options.help = true;
|
||||
continue;
|
||||
}
|
||||
const next = argv[index + 1];
|
||||
if (!next || next.startsWith("--")) {
|
||||
throw new Error(`${arg} requires a value`);
|
||||
}
|
||||
index += 1;
|
||||
if (arg === "--repo-root") options.repoRoot = next;
|
||||
else if (arg === "--events-file") options.eventsFile = next;
|
||||
else if (arg === "--repo") options.repo = next;
|
||||
else if (arg === "--ref") options.ref = next;
|
||||
else if (arg === "--base-ref") options.baseRef = next;
|
||||
else throw new Error(`unknown option: ${arg}`);
|
||||
}
|
||||
return options;
|
||||
}
|
||||
|
||||
function printHelp() {
|
||||
process.stdout.write(`Usage: node scripts/extract-proposed-events.mjs [options]\n\nOptions:\n --repo-root <path> Repository root. Defaults to cwd.\n --events-file <path> events.ts path, absolute or repo-relative.\n --repo <slug> Source repository slug. Defaults to paperclipai/paperclip.\n --ref <ref> Source ref/SHA for the extractor envelope.\n --base-ref <ref> Optional base ref for diff-oriented inventory jobs.\n`);
|
||||
}
|
||||
|
||||
if (process.argv[1] === fileURLToPath(import.meta.url)) {
|
||||
try {
|
||||
const options = parseArgs(process.argv.slice(2));
|
||||
if (options.help) {
|
||||
printHelp();
|
||||
process.exit(0);
|
||||
}
|
||||
process.stdout.write(`${JSON.stringify(extractProposedEvents(options), null, 2)}\n`);
|
||||
} catch (error) {
|
||||
process.stderr.write(`${error instanceof Error ? error.message : String(error)}\n`);
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,261 @@
|
|||
import assert from "node:assert/strict";
|
||||
import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import test from "node:test";
|
||||
import ts from "typescript";
|
||||
|
||||
import {
|
||||
PROPOSED_TELEMETRY_SCHEMA_VERSION,
|
||||
assertRepoRelativePath,
|
||||
extractProposedEvents,
|
||||
toRepoRelativePath,
|
||||
} from "./extract-proposed-events.mjs";
|
||||
|
||||
function withFixtureRepo(source, callback) {
|
||||
const repoRoot = mkdtempSync(join(tmpdir(), "paperclip-proposed-events-"));
|
||||
const eventsFile = join(repoRoot, "packages", "shared", "src", "telemetry", "events.ts");
|
||||
mkdirSync(join(repoRoot, "packages", "shared", "src", "telemetry"), { recursive: true });
|
||||
writeFileSync(eventsFile, source);
|
||||
try {
|
||||
return callback({ repoRoot, eventsFile });
|
||||
} finally {
|
||||
rmSync(repoRoot, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
const fixtureSource = `
|
||||
import type { TelemetryClient } from "./client.js";
|
||||
|
||||
type RawDimension<T extends string | undefined> = T | (string & {});
|
||||
|
||||
export function trackSkillStudioCreated(
|
||||
client: TelemetryClient,
|
||||
dims: {
|
||||
sharing_scope: RawDimension<"team" | "private">;
|
||||
category_count: number;
|
||||
launched_from_template: boolean;
|
||||
},
|
||||
): void {
|
||||
client.track(
|
||||
// @ts-expect-error -- proposed-telemetry(PAP-2411): measure Skill Studio create completion
|
||||
"skill_studio.skill_created",
|
||||
dims,
|
||||
);
|
||||
}
|
||||
|
||||
export function trackSkillStudioOpened(
|
||||
client: TelemetryClient,
|
||||
dims: {
|
||||
surface: "modal" | "page";
|
||||
},
|
||||
): void {
|
||||
client.track(
|
||||
// @ts-expect-error
|
||||
"skill_studio.opened",
|
||||
dims,
|
||||
);
|
||||
}
|
||||
|
||||
export function trackInstallStarted(client: TelemetryClient): void {
|
||||
client.track("install.started", {});
|
||||
}
|
||||
`;
|
||||
|
||||
test("extractor emits deterministic proposed-telemetry-extractor.v2 records", () => {
|
||||
const output = withFixtureRepo(fixtureSource, ({ repoRoot, eventsFile }) =>
|
||||
extractProposedEvents({ repoRoot, eventsFile, ref: "fixture-sha", baseRef: "master" }),
|
||||
);
|
||||
|
||||
assert.equal(output.schemaVersion, PROPOSED_TELEMETRY_SCHEMA_VERSION);
|
||||
assert.deepEqual(output.source, {
|
||||
repo: "paperclipai/paperclip",
|
||||
ref: "fixture-sha",
|
||||
baseRef: "master",
|
||||
});
|
||||
assert.deepEqual(
|
||||
output.proposals.map((proposal) => proposal.name),
|
||||
["skill_studio.opened", "skill_studio.skill_created"],
|
||||
);
|
||||
|
||||
const created = output.proposals.find((proposal) => proposal.name === "skill_studio.skill_created");
|
||||
assert.deepEqual(created.dimensions, [
|
||||
{ name: "category_count", type: "number" },
|
||||
{ name: "launched_from_template", type: "boolean" },
|
||||
{ name: "sharing_scope", type: "string" },
|
||||
]);
|
||||
assert.deepEqual(created.rationale, {
|
||||
issue: "PAP-2411",
|
||||
text: "measure Skill Studio create completion",
|
||||
missingIssue: false,
|
||||
missingRationale: false,
|
||||
});
|
||||
assert.equal(created.provenance.length, 1);
|
||||
assert.equal(created.provenance[0].file, "packages/shared/src/telemetry/events.ts");
|
||||
assert.equal(typeof created.provenance[0].line, "number");
|
||||
assert.equal(typeof created.provenance[0].column, "number");
|
||||
});
|
||||
|
||||
test("extractor flags a missing proposed-telemetry suffix without hard-failing", () => {
|
||||
const output = withFixtureRepo(fixtureSource, ({ repoRoot, eventsFile }) =>
|
||||
extractProposedEvents({ repoRoot, eventsFile, ref: "fixture-sha" }),
|
||||
);
|
||||
|
||||
const opened = output.proposals.find((proposal) => proposal.name === "skill_studio.opened");
|
||||
assert.deepEqual(opened.rationale, {
|
||||
issue: null,
|
||||
text: null,
|
||||
missingIssue: true,
|
||||
missingRationale: true,
|
||||
});
|
||||
assert.deepEqual(opened.dimensions, [{ name: "surface", type: "string" }]);
|
||||
});
|
||||
|
||||
test("extractor scans TelemetryClient wrappers whose receiver is not named client", () => {
|
||||
const output = withFixtureRepo(
|
||||
`type TelemetryClient = { track(name: string, dims: unknown): void };
|
||||
|
||||
export function trackWorkspaceOpened(
|
||||
telemetry: TelemetryClient,
|
||||
dims: { surface: string },
|
||||
): void {
|
||||
telemetry.track(
|
||||
// @ts-expect-error -- proposed-telemetry(PAP-2463): exercise alternate telemetry client parameter names
|
||||
"workspace.opened",
|
||||
dims,
|
||||
);
|
||||
}
|
||||
`,
|
||||
({ repoRoot, eventsFile }) => extractProposedEvents({ repoRoot, eventsFile, ref: "fixture-sha" }),
|
||||
);
|
||||
|
||||
assert.deepEqual(
|
||||
output.proposals.map((proposal) => proposal.name),
|
||||
["workspace.opened"],
|
||||
);
|
||||
assert.deepEqual(output.proposals[0].dimensions, [{ name: "surface", type: "string" }]);
|
||||
});
|
||||
|
||||
test("extractor scans variable-assigned wrappers and ignores nullable union members", () => {
|
||||
const output = withFixtureRepo(
|
||||
`type TelemetryClient = { track(name: string, dims: unknown): void };
|
||||
|
||||
export const trackWorkspaceArrow = (
|
||||
telemetry: TelemetryClient,
|
||||
dims: { surface: string | null | undefined },
|
||||
): void => {
|
||||
telemetry.track(
|
||||
// @ts-expect-error -- proposed-telemetry(PAP-2463): exercise arrow wrapper extraction
|
||||
"workspace.arrow_opened",
|
||||
dims,
|
||||
);
|
||||
};
|
||||
|
||||
export const trackWorkspaceFunctionExpression = function (
|
||||
tc: TelemetryClient,
|
||||
dims: { accepted: true | false | null },
|
||||
): void {
|
||||
tc.track(
|
||||
// @ts-expect-error -- proposed-telemetry(PAP-2463): exercise function-expression wrapper extraction
|
||||
"workspace.function_expression_opened",
|
||||
dims,
|
||||
);
|
||||
};
|
||||
`,
|
||||
({ repoRoot, eventsFile }) => extractProposedEvents({ repoRoot, eventsFile, ref: "fixture-sha" }),
|
||||
);
|
||||
|
||||
assert.deepEqual(
|
||||
output.proposals.map((proposal) => proposal.name),
|
||||
["workspace.arrow_opened", "workspace.function_expression_opened"],
|
||||
);
|
||||
assert.deepEqual(output.proposals[0].dimensions, [{ name: "surface", type: "string" }]);
|
||||
assert.deepEqual(output.proposals[1].dimensions, [{ name: "accepted", type: "boolean" }]);
|
||||
});
|
||||
|
||||
test("extractor rejects invalid rationale issue references when present", () => {
|
||||
assert.throws(
|
||||
() =>
|
||||
withFixtureRepo(
|
||||
`export function trackBad(client, dims: { source: string }): void {\n client.track(\n // @ts-expect-error -- proposed-telemetry(PROJ-1): bad issue ref\n "skill_studio.bad_issue",\n dims,\n );\n}\n`,
|
||||
({ repoRoot, eventsFile }) => extractProposedEvents({ repoRoot, eventsFile, ref: "fixture-sha" }),
|
||||
),
|
||||
/rationale issue must be PAP-<digits>/,
|
||||
);
|
||||
});
|
||||
|
||||
test("provenance paths are repo-relative and reject dev-host path shapes", () => {
|
||||
assert.equal(
|
||||
assertRepoRelativePath("packages/shared/src/telemetry/events.ts"),
|
||||
"packages/shared/src/telemetry/events.ts",
|
||||
);
|
||||
assert.throws(() => assertRepoRelativePath("/tmp/events.ts"), /repo-relative/);
|
||||
assert.throws(() => assertRepoRelativePath("../events.ts"), /unsafe path segment/);
|
||||
assert.throws(() => assertRepoRelativePath("C:/repo/events.ts"), /drive-letter/);
|
||||
assert.throws(() => assertRepoRelativePath("packages\\shared\\events.ts"), /forward slashes/);
|
||||
|
||||
const repoRoot = mkdtempSync(join(tmpdir(), "paperclip-provenance-root-"));
|
||||
try {
|
||||
assert.throws(() => toRepoRelativePath(repoRoot, join(repoRoot, "..", "events.ts")), /inside repo root/);
|
||||
} finally {
|
||||
rmSync(repoRoot, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
function diagnosticsFor(sourceText) {
|
||||
const repoRoot = mkdtempSync(join(tmpdir(), "paperclip-ts2578-"));
|
||||
const fileName = join(repoRoot, "fixture.ts");
|
||||
writeFileSync(fileName, sourceText);
|
||||
try {
|
||||
const program = ts.createProgram([fileName], {
|
||||
strict: true,
|
||||
noEmit: true,
|
||||
skipLibCheck: true,
|
||||
target: ts.ScriptTarget.ES2023,
|
||||
module: ts.ModuleKind.NodeNext,
|
||||
moduleResolution: ts.ModuleResolutionKind.NodeNext,
|
||||
types: [],
|
||||
});
|
||||
return ts
|
||||
.getPreEmitDiagnostics(program)
|
||||
.filter((diagnostic) => diagnostic.file?.fileName === fileName)
|
||||
.map((diagnostic) => diagnostic.code);
|
||||
} finally {
|
||||
rmSync(repoRoot, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
function tsMechanicsFixture(eventUnion, mapEntry) {
|
||||
return `
|
||||
type TelemetryEventName = ${eventUnion};
|
||||
interface EventDimensionsMap {
|
||||
"install.started": {};
|
||||
${mapEntry}
|
||||
}
|
||||
type TelemetryEventDimensions<K extends TelemetryEventName> = EventDimensionsMap[K];
|
||||
type TrackArgs<K extends TelemetryEventName> = keyof TelemetryEventDimensions<K> extends never
|
||||
? [dimensions?: TelemetryEventDimensions<K>]
|
||||
: [dimensions: TelemetryEventDimensions<K>];
|
||||
declare const client: {
|
||||
track<K extends TelemetryEventName>(eventName: K, ...args: TrackArgs<K>): void;
|
||||
};
|
||||
client.track(
|
||||
// @ts-expect-error -- proposed-telemetry(PAP-2411): TS2578 expiry fixture
|
||||
"skill_studio.skill_created",
|
||||
{ sharing_scope: "team" },
|
||||
);
|
||||
`;
|
||||
}
|
||||
|
||||
test("TS2578 expires the directive once a fixture event is registered", () => {
|
||||
const unregistered = diagnosticsFor(tsMechanicsFixture('"install.started"', ""));
|
||||
assert.deepEqual(unregistered, []);
|
||||
|
||||
const registered = diagnosticsFor(
|
||||
tsMechanicsFixture(
|
||||
'"install.started" | "skill_studio.skill_created"',
|
||||
'"skill_studio.skill_created": { sharing_scope: string };',
|
||||
),
|
||||
);
|
||||
assert.ok(registered.includes(2578), `expected TS2578, got ${registered.join(", ")}`);
|
||||
});
|
||||
Loading…
Reference in New Issue