From e68ee098091611ee450e88d23aadaf3522d825a2 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Wed, 8 Jul 2026 08:04:39 -0700 Subject: [PATCH] perf(release): batch npm registry version queries (#9202) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The release subsystem publishes the public workspace packages and also powers release-related CI validation. > - The release flow currently asks npm for package versions one package at a time in multiple places. > - That serial registry latency slows the PR Canary Dry Run path and real release invocations even though the checks are independent. > - This pull request batches npm registry version lookups with bounded concurrency and reuses the result for version calculation. > - The benefit is shorter non-build release-script time while preserving the fresh target-version existence check before publishing. ## Linked Issues or Issue Description - No public GitHub issue exists for this release-script performance cleanup. ### Problem or motivation Release validation spends avoidable time on repeated serial `npm view` calls across the public package set. The slow path affects PR release validation and real release invocations because version discovery waits on independent registry reads one at a time. ### Proposed solution Fetch package version maps concurrently with bounded parallelism, reuse that map for stable/canary version calculation, and keep a fresh parallel absence check for the target publish version. ### Alternatives considered Keeping the existing serial shell loop is simpler, but it preserves the CI latency cost. Caching the final target-version existence check was rejected because release publish safety should still query npm freshly before publishing. ### Roadmap alignment This is a small release-tooling performance improvement. It does not duplicate any planned core product work found in `ROADMAP.md`. ## What Changed - Added `scripts/release-registry-versions.mjs` to fetch npm package version maps and assert target-version absence with bounded parallelism. - Updated `scripts/release.sh` to prefetch package versions once and to batch the final target-version absence check. - Updated `next_stable_version` and `next_canary_version` to use the prefetched version map when present, with the existing per-package npm fallback preserved. - Added release-registry helper coverage and included it in `pnpm run test:release-registry`. - Hardened the release publish helper tests so their fake `pnpm`/`npm` fixture PATH is preserved under non-login shell execution. ## Verification - `node --test scripts/release-registry-versions.test.mjs` - `pnpm run test:release-registry` - `bash -n scripts/release.sh scripts/release-lib.sh` - `git diff --check` - Safety scan before push: searched changed files for common key/token/password patterns and PII markers; only benign script-name text matched (`secrets:migrate-inline-env`). - Remote PR checks on the latest head passed, including `Typecheck + Release Registry`, `Canary Dry Run`, build, tests, e2e, policy, security scans, and commitperclip review. - Greptile reviewed the latest head with Confidence Score 5/5 and no blocking issues. ## Risks - Low risk. The release version helpers keep their original npm fallback when no prefetched version map is supplied. - The existence check remains fresh and uncached before publish, but now reports all matching package/version pairs from a parallel check. - If npm has transient failures during the prefetch step, missing or failed packages still map to an empty version list, matching the old helper behavior. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex coding agent using GPT-5, with shell/tool execution in the local repository. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Claude --- package.json | 2 +- scripts/release-lib.sh | 74 +++++--- scripts/release-lib.test.mjs | 3 +- scripts/release-registry-versions.mjs | 114 ++++++++++++ scripts/release-registry-versions.test.mjs | 197 +++++++++++++++++++++ scripts/release.sh | 20 ++- 6 files changed, 381 insertions(+), 29 deletions(-) create mode 100644 scripts/release-registry-versions.mjs create mode 100644 scripts/release-registry-versions.test.mjs diff --git a/package.json b/package.json index b9f040ef7e..89ef88fd09 100644 --- a/package.json +++ b/package.json @@ -48,7 +48,7 @@ "smoke:openclaw-sse-standalone": "./scripts/smoke/openclaw-sse-standalone.sh", "smoke:pipelines-tutorial": "./scripts/smoke/pipelines-tutorial-smoke.sh", "smoke:terminal-bench-loop-skill": "node scripts/smoke/terminal-bench-loop-skill-smoke.mjs", - "test:release-registry": "node --test scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/link-plugin-dev-sdk.test.js", + "test:release-registry": "node --test scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js", "storybook-visual:baseline": "node scripts/storybook-visual-baseline.mjs", "test:storybook-visual": "node scripts/storybook-visual-baseline.mjs download && node scripts/storybook-visual-baseline.mjs verify && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts", "test:storybook-visual:update": "node scripts/storybook-visual-baseline.mjs download && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts --update-snapshots && node scripts/storybook-visual-baseline.mjs pack", diff --git a/scripts/release-lib.sh b/scripts/release-lib.sh index 240725e3c9..9fbbfdbaf9 100644 --- a/scripts/release-lib.sh +++ b/scripts/release-lib.sh @@ -142,6 +142,7 @@ next_stable_version() { const input = process.argv[2]; const packageNames = process.argv.slice(3); const { execSync } = require("node:child_process"); +const { readFileSync } = require("node:fs"); const date = input ? new Date(`${input}T00:00:00Z`) : new Date(); if (Number.isNaN(date.getTime())) { @@ -149,6 +150,17 @@ if (Number.isNaN(date.getTime())) { process.exit(1); } +// Optional pre-fetched version data (see release-registry-versions.mjs). +// Avoids one serial `npm view` round-trip per package. +let versionsCache = null; +if (process.env.RELEASE_PACKAGE_VERSIONS_FILE) { + try { + versionsCache = JSON.parse(readFileSync(process.env.RELEASE_PACKAGE_VERSIONS_FILE, "utf8")); + } catch { + versionsCache = null; + } +} + const stableSlot = `${date.getUTCFullYear()}.${date.getUTCMonth() + 1}${String(date.getUTCDate()).padStart(2, "0")}`; const pattern = new RegExp(`^${stableSlot.replace(/\./g, '\\.')}\.(\\d+)$`); let max = -1; @@ -156,18 +168,22 @@ let max = -1; for (const packageName of packageNames) { let versions = []; - try { - const raw = execSync(`npm view ${JSON.stringify(packageName)} versions --json`, { - encoding: "utf8", - stdio: ["ignore", "pipe", "ignore"], - }).trim(); + if (versionsCache && Array.isArray(versionsCache[packageName])) { + versions = versionsCache[packageName]; + } else { + try { + const raw = execSync(`npm view ${JSON.stringify(packageName)} versions --json`, { + encoding: "utf8", + stdio: ["ignore", "pipe", "ignore"], + }).trim(); - if (raw) { - const parsed = JSON.parse(raw); - versions = Array.isArray(parsed) ? parsed : [parsed]; + if (raw) { + const parsed = JSON.parse(raw); + versions = Array.isArray(parsed) ? parsed : [parsed]; + } + } catch { + versions = []; } - } catch { - versions = []; } for (const version of versions) { @@ -189,6 +205,18 @@ next_canary_version() { const stable = process.argv[2]; const packageNames = process.argv.slice(3); const { execSync } = require("node:child_process"); +const { readFileSync } = require("node:fs"); + +// Optional pre-fetched version data (see release-registry-versions.mjs). +// Avoids one serial `npm view` round-trip per package. +let versionsCache = null; +if (process.env.RELEASE_PACKAGE_VERSIONS_FILE) { + try { + versionsCache = JSON.parse(readFileSync(process.env.RELEASE_PACKAGE_VERSIONS_FILE, "utf8")); + } catch { + versionsCache = null; + } +} const pattern = new RegExp(`^${stable.replace(/\./g, '\\.')}-canary\\.(\\d+)$`); let max = -1; @@ -196,20 +224,24 @@ let max = -1; for (const packageName of packageNames) { let versions = []; - try { - const raw = execSync(`npm view ${JSON.stringify(packageName)} versions --json`, { - encoding: "utf8", - stdio: ["ignore", "pipe", "ignore"], - }).trim(); + if (versionsCache && Array.isArray(versionsCache[packageName])) { + versions = versionsCache[packageName]; + } else { + try { + const raw = execSync(`npm view ${JSON.stringify(packageName)} versions --json`, { + encoding: "utf8", + stdio: ["ignore", "pipe", "ignore"], + }).trim(); - if (raw) { - const parsed = JSON.parse(raw); - versions = Array.isArray(parsed) ? parsed : [parsed]; + if (raw) { + const parsed = JSON.parse(raw); + versions = Array.isArray(parsed) ? parsed : [parsed]; + } + } catch { + versions = []; } - } catch { - versions = []; } - + for (const version of versions) { const match = version.match(pattern); if (!match) continue; diff --git a/scripts/release-lib.test.mjs b/scripts/release-lib.test.mjs index fcaf52e6d1..677c6e5270 100644 --- a/scripts/release-lib.test.mjs +++ b/scripts/release-lib.test.mjs @@ -18,6 +18,7 @@ function runPublishHelper({ pnpmMode, npmVersionExists = false, distTag = "canar const callLog = join(fixtureDir, "calls.log"); mkdirSync(binDir); mkdirSync(stateDir); + writeFileSync(callLog, ""); writeExecutable( join(binDir, "pnpm"), @@ -84,7 +85,7 @@ publish_package_to_npm ${distTag} @paperclipai/example 1.2.3 let status = 0; let output = ""; try { - output = execFileSync("bash", ["-lc", script], { + output = execFileSync("bash", ["-c", script], { cwd: fixtureDir, encoding: "utf8", env: { diff --git a/scripts/release-registry-versions.mjs b/scripts/release-registry-versions.mjs new file mode 100644 index 0000000000..7cf7436e22 --- /dev/null +++ b/scripts/release-registry-versions.mjs @@ -0,0 +1,114 @@ +#!/usr/bin/env node +// Batched npm registry version queries for the release tooling. +// +// The release flow needs published-version data for every public workspace +// package. Querying them one `npm view` at a time is serial network latency +// that dominates the non-build time of `release.sh` (and the PR workflow's +// Canary Dry Run job). This helper runs the same `npm view` queries with +// bounded concurrency instead. +// +// Usage: +// node scripts/release-registry-versions.mjs fetch +// Prints a JSON object mapping each package name to its published +// versions array. Packages that are missing from the registry (or fail +// to resolve) map to []. +// +// node scripts/release-registry-versions.mjs assert-absent +// Freshly checks that is not published for any . Exits 0 +// when absent everywhere; prints the offending package@version pairs to +// stderr and exits 1 otherwise. + +import { execFile } from "node:child_process"; + +const CONCURRENCY = Number(process.env.RELEASE_REGISTRY_CONCURRENCY || 10); +if (!Number.isInteger(CONCURRENCY) || CONCURRENCY < 1) { + console.error("RELEASE_REGISTRY_CONCURRENCY must be a positive integer."); + process.exit(2); +} + +function npmView(args) { + return new Promise((resolve) => { + execFile("npm", ["view", ...args], { encoding: "utf8" }, (error, stdout) => { + if (error) { + resolve(null); + return; + } + resolve(stdout.trim()); + }); + }); +} + +async function mapWithConcurrency(items, limit, fn) { + const results = new Array(items.length); + let next = 0; + + async function worker() { + while (next < items.length) { + const index = next; + next += 1; + results[index] = await fn(items[index]); + } + } + + const workers = []; + for (let i = 0; i < Math.min(limit, items.length); i += 1) { + workers.push(worker()); + } + await Promise.all(workers); + return results; +} + +async function fetchVersions(packageNames) { + const versionLists = await mapWithConcurrency(packageNames, CONCURRENCY, async (packageName) => { + const raw = await npmView([packageName, "versions", "--json"]); + if (!raw) return []; + try { + const parsed = JSON.parse(raw); + return Array.isArray(parsed) ? parsed : [parsed]; + } catch { + return []; + } + }); + + const map = {}; + packageNames.forEach((packageName, index) => { + map[packageName] = versionLists[index]; + }); + return map; +} + +async function assertAbsent(version, packageNames) { + const resolved = await mapWithConcurrency(packageNames, CONCURRENCY, async (packageName) => { + const raw = await npmView([`${packageName}@${version}`, "version"]); + return raw === version ? packageName : null; + }); + + return resolved.filter((packageName) => packageName !== null); +} + +const [mode, ...rest] = process.argv.slice(2); + +if (mode === "fetch") { + if (rest.length === 0) { + console.error("usage: release-registry-versions.mjs fetch "); + process.exit(2); + } + const map = await fetchVersions(rest); + process.stdout.write(`${JSON.stringify(map)}\n`); +} else if (mode === "assert-absent") { + const [version, ...packageNames] = rest; + if (!version || packageNames.length === 0) { + console.error("usage: release-registry-versions.mjs assert-absent "); + process.exit(2); + } + const existing = await assertAbsent(version, packageNames); + if (existing.length > 0) { + for (const packageName of existing) { + console.error(`npm version ${packageName}@${version} already exists.`); + } + process.exit(1); + } +} else { + console.error("usage: release-registry-versions.mjs ..."); + process.exit(2); +} diff --git a/scripts/release-registry-versions.test.mjs b/scripts/release-registry-versions.test.mjs new file mode 100644 index 0000000000..2915dae665 --- /dev/null +++ b/scripts/release-registry-versions.test.mjs @@ -0,0 +1,197 @@ +import assert from "node:assert/strict"; +import { execFileSync } from "node:child_process"; +import { mkdirSync, mkdtempSync, readFileSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import test from "node:test"; + +const repoRoot = new URL("..", import.meta.url).pathname.replace(/\/$/, ""); +const scriptPath = join(repoRoot, "scripts", "release-registry-versions.mjs"); + +function writeExecutable(path, body) { + writeFileSync(path, body, { mode: 0o755 }); +} + +function makeFixture() { + const fixtureDir = mkdtempSync(join(tmpdir(), "paperclip-release-registry-")); + const binDir = join(fixtureDir, "bin"); + const callLog = join(fixtureDir, "calls.log"); + mkdirSync(binDir); + writeFileSync(callLog, ""); + + writeExecutable( + join(binDir, "npm"), + `#!/usr/bin/env bash +set -euo pipefail +printf 'npm %s\\n' "$*" >> "$FAKE_CALL_LOG" +target="$2" +case "$target" in + "@paperclipai/present@"*) + printf '%s\\n' "\${target##*@}" + ;; + "@paperclipai/absent@"*) + exit 1 + ;; + "@paperclipai/present") + echo '["1.0.0","2026.707.0","2026.707.1","2026.707.1-canary.4"]' + ;; + *) + exit 1 + ;; +esac +`, + ); + + return { fixtureDir, binDir, callLog }; +} + +function runScript(args, { binDir, callLog }, extraEnv = {}) { + let status = 0; + let stdout = ""; + let stderr = ""; + try { + stdout = execFileSync("node", [scriptPath, ...args], { + encoding: "utf8", + env: { + ...process.env, + PATH: `${binDir}:${process.env.PATH}`, + FAKE_CALL_LOG: callLog, + ...extraEnv, + }, + stdio: ["ignore", "pipe", "pipe"], + }); + } catch (error) { + status = error.status ?? 1; + stdout = error.stdout ?? ""; + stderr = error.stderr ?? ""; + } + return { status, stdout, stderr, calls: readFileSync(callLog, "utf8") }; +} + +function runReleaseLibHelper(fnCall, { binDir, callLog }, extraEnv = {}) { + const script = ` +set -euo pipefail +source "${repoRoot}/scripts/release-lib.sh" +${fnCall} +`; + let status = 0; + let output = ""; + try { + output = execFileSync("bash", ["-c", script], { + encoding: "utf8", + env: { + ...process.env, + PATH: `${binDir}:${process.env.PATH}`, + FAKE_CALL_LOG: callLog, + REPO_ROOT: repoRoot, + ...extraEnv, + }, + stdio: ["ignore", "pipe", "pipe"], + }); + } catch (error) { + status = error.status ?? 1; + output = `${error.stdout ?? ""}${error.stderr ?? ""}`; + } + return { status, output, calls: readFileSync(callLog, "utf8") }; +} + +test("fetch prints a JSON version map and treats missing packages as empty", () => { + const fixture = makeFixture(); + const result = runScript(["fetch", "@paperclipai/present", "@paperclipai/missing"], fixture); + + assert.equal(result.status, 0); + const map = JSON.parse(result.stdout); + assert.deepEqual(map["@paperclipai/present"], [ + "1.0.0", + "2026.707.0", + "2026.707.1", + "2026.707.1-canary.4", + ]); + assert.deepEqual(map["@paperclipai/missing"], []); + assert.match(result.calls, /^npm view @paperclipai\/present versions --json$/m); + assert.match(result.calls, /^npm view @paperclipai\/missing versions --json$/m); +}); + +test("assert-absent succeeds when no package has the version", () => { + const fixture = makeFixture(); + const result = runScript( + ["assert-absent", "2026.707.2", "@paperclipai/absent", "@paperclipai/absent"], + fixture, + ); + + assert.equal(result.status, 0); + assert.match(result.calls, /^npm view @paperclipai\/absent@2026\.707\.2 version$/m); +}); + +test("assert-absent fails and names packages that already have the version", () => { + const fixture = makeFixture(); + const result = runScript( + ["assert-absent", "2026.707.2", "@paperclipai/present", "@paperclipai/absent"], + fixture, + ); + + assert.equal(result.status, 1); + assert.match(result.stderr, /npm version @paperclipai\/present@2026\.707\.2 already exists\./); + assert.doesNotMatch(result.stderr, /@paperclipai\/absent@/); +}); + +test("invalid concurrency fails instead of skipping registry checks", () => { + const fixture = makeFixture(); + const result = runScript(["assert-absent", "2026.707.2", "@paperclipai/present"], fixture, { + RELEASE_REGISTRY_CONCURRENCY: "0", + }); + + assert.equal(result.status, 2); + assert.match(result.stderr, /RELEASE_REGISTRY_CONCURRENCY must be a positive integer\./); + assert.equal(result.calls, ""); +}); + +test("next_stable_version reads RELEASE_PACKAGE_VERSIONS_FILE without calling npm", () => { + const fixture = makeFixture(); + const versionsFile = join(fixture.fixtureDir, "versions.json"); + writeFileSync( + versionsFile, + JSON.stringify({ + "@paperclipai/a": ["2026.707.0", "2026.707.1", "2026.707.1-canary.4"], + "@paperclipai/b": [], + }), + ); + + const result = runReleaseLibHelper( + 'next_stable_version 2026-07-07 "@paperclipai/a" "@paperclipai/b"', + fixture, + { RELEASE_PACKAGE_VERSIONS_FILE: versionsFile }, + ); + + assert.equal(result.status, 0); + assert.equal(result.output, "2026.707.2"); + assert.doesNotMatch(result.calls, /npm view/); +}); + +test("next_canary_version reads RELEASE_PACKAGE_VERSIONS_FILE without calling npm", () => { + const fixture = makeFixture(); + const versionsFile = join(fixture.fixtureDir, "versions.json"); + writeFileSync( + versionsFile, + JSON.stringify({ + "@paperclipai/a": ["2026.707.0", "2026.707.1", "2026.707.1-canary.4"], + }), + ); + + const result = runReleaseLibHelper('next_canary_version 2026.707.1 "@paperclipai/a"', fixture, { + RELEASE_PACKAGE_VERSIONS_FILE: versionsFile, + }); + + assert.equal(result.status, 0); + assert.equal(result.output, "2026.707.1-canary.5"); + assert.doesNotMatch(result.calls, /npm view/); +}); + +test("next_stable_version falls back to npm view without a versions file", () => { + const fixture = makeFixture(); + const result = runReleaseLibHelper('next_stable_version 2026-07-07 "@paperclipai/present"', fixture); + + assert.equal(result.status, 0); + assert.equal(result.output, "2026.707.2"); + assert.match(result.calls, /^npm view @paperclipai\/present versions --json$/m); +}); diff --git a/scripts/release.sh b/scripts/release.sh index aa394e91f0..ed069d92a5 100755 --- a/scripts/release.sh +++ b/scripts/release.sh @@ -133,6 +133,13 @@ done < <(printf '%s\n' "$PUBLIC_PACKAGE_INFO" | cut -f2) [ -n "$PUBLIC_PACKAGE_INFO" ] || release_fail "no public packages were found in the workspace." +# Pre-fetch published versions for every public package in parallel so the +# version helpers below do not each issue one serial `npm view` call per +# package (see scripts/release-registry-versions.mjs). +RELEASE_PACKAGE_VERSIONS_FILE="$(mktemp)" +export RELEASE_PACKAGE_VERSIONS_FILE +node "$REPO_ROOT/scripts/release-registry-versions.mjs" fetch "${PUBLIC_PACKAGE_NAMES[@]}" > "$RELEASE_PACKAGE_VERSIONS_FILE" + TARGET_STABLE_VERSION="$(next_stable_version "$RELEASE_DATE" "${PUBLIC_PACKAGE_NAMES[@]}")" TARGET_PUBLISH_VERSION="$TARGET_STABLE_VERSION" DIST_TAG="latest" @@ -146,6 +153,9 @@ else tag_name="$(stable_tag_name "$TARGET_STABLE_VERSION")" fi +rm -f "$RELEASE_PACKAGE_VERSIONS_FILE" +unset RELEASE_PACKAGE_VERSIONS_FILE + if [ "$print_version_only" = true ]; then printf '%s\n' "$TARGET_PUBLISH_VERSION" exit 0 @@ -168,12 +178,10 @@ if git_local_tag_exists "$tag_name" || git_remote_tag_exists "$tag_name" "$PUBLI release_fail "git tag $tag_name already exists locally or on $PUBLISH_REMOTE." fi -while IFS= read -r package_name; do - [ -z "$package_name" ] && continue - if npm_package_version_exists "$package_name" "$TARGET_PUBLISH_VERSION"; then - release_fail "npm version ${package_name}@${TARGET_PUBLISH_VERSION} already exists." - fi -done <<< "$(printf '%s\n' "${PUBLIC_PACKAGE_NAMES[@]}")" +# Fresh (non-cached) existence check, batched in parallel. Prints the +# offending package@version pairs itself before failing. +node "$REPO_ROOT/scripts/release-registry-versions.mjs" assert-absent "$TARGET_PUBLISH_VERSION" "${PUBLIC_PACKAGE_NAMES[@]}" \ + || release_fail "npm version ${TARGET_PUBLISH_VERSION} already exists for one or more packages." release_info "" release_info "==> Release plan"