From 204c416478a607601c9c6caff419ca31f30b50ef Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 22 Jul 2026 17:01:11 -0500 Subject: [PATCH] fix(release): publish bundled packages with trusted npm staging (#10047) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Paperclip publishes its CLI, server, adapters, and shared packages through automated canary and stable release workflows. > - `@paperclipai/adapter-utils` bundles the patched `acpx` runtime, so it must use npm 11 for OIDC trusted publishing. > - The prior staging directory contained pnpm's `.pnpm` symlink forest, which crashes npm 11's directory-pack step on GitHub runners and produces a consumer-broken bundled dependency tree. > - This pull request rebuilds staged production dependencies as a physical npm tree, reapplies repository patches, and publishes that clean directory directly with npm 11 trusted publishing. > - The benefit is a release path that retains GitHub Actions OIDC trusted publishing while shipping a working patched acpx runtime to consumers. ## Linked Issues or Issue Description Refs: #9980, #10030, #10041 No public GitHub issue exists for this release failure. ### What happened? Canary and stable publishing began routing `@paperclipai/adapter-utils` through npm after it declared `bundleDependencies: ["acpx"]`. Publishing the pnpm-deployed directory with npm 11 crashes during npm's directory-pack phase on GitHub runners with `Exit handler never called!`. The same staged shape also produces a broken consumer artifact because acpx cannot resolve transitive runtime dependencies after installation. ### Expected behavior Bundled packages publish directly from a self-contained staging directory through npm 11 OIDC trusted publishing, and consumers receive a working patched acpx runtime with its transitive dependencies. ### Steps to reproduce 1. Stage `packages/adapter-utils` using the old `pnpm deploy`-only shape. 2. Publish that directory with npm 11 on a GitHub runner. 3. npm crashes before registry/OIDC activity while walking the `.pnpm` symlink forest. 4. Install an artifact packed from that old shape into a fresh npm project and run acpx; its runtime dependency resolution fails. ### Deployment mode GitHub Actions canary/stable release workflow. ### Relevant logs or output ```text npm error Exit handler never called! ``` ## What Changed - After `pnpm deploy`, remove the staged pnpm `node_modules` tree and run `npm install --omit=dev --ignore-scripts --no-audit --no-fund` to create a physical hoisted production tree. - Apply every root `pnpm.patchedDependencies` patch whose package is declared in the staged package's bundled dependencies, failing staging if any patch cannot apply. - Assert the staged acpx runtime contains the required `onAgentStderr` patch marker. - Publish the clean staging directory directly with pinned npm 11.18.0, retaining GitHub Actions OIDC trusted publishing, verbose diagnostics, and the duplicate-transparency-log retry without provenance. - Keep pinned npm 10.9.7 packing only for local/dry-run payload verification; registry publishing does not use a tarball argument. - Add focused coverage for npm-tree staging, patch application, direct directory publish arguments, and bundled tlog retries. ## Verification - `bash -n scripts/release-lib.sh scripts/release.sh` - `node --test scripts/release-lib.test.mjs scripts/acpx-patch-packaging.test.mjs` — 12/12 passed. - `pnpm test:release-registry` — 68/68 passed. - Real staging smoke: `node scripts/prepare-bundled-package.mjs packages/adapter-utils ` produced a real `node_modules/acpx` directory, no `.pnpm` directory, and the `onAgentStderr` patch marker. - Real npm 11 directory-publish smoke: `npx --yes npm@11.18.0 publish --dry-run --tag canary --access public --loglevel verbose` packed 26 bundled dependencies and reached the expected existing-version registry rejection without `Exit handler never called!`. - The merge-triggered `publish_canary` workflow remains the live OIDC trusted-publishing verification. ## Risks - The live GitHub Actions trusted-publishing path can only be fully proven by the merge-triggered canary run; npm debug-log upload remains available if it fails. - Bundling acpx continues to freeze platform-specific transitive artifacts such as esbuild binaries from the Linux release runner. This is a pre-existing consequence of the bundling decision in #9980 and is not expanded here. - Rebuilding dependencies with npm depends on the exact bundled dependency versions in the staged manifest; staging fails hard if repository patches no longer apply. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex, exact model ID `gpt-5.6-sol`, high reasoning mode, with repository tool use and code execution. The harness did not expose a model context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- scripts/acpx-patch-packaging.test.mjs | 95 +++++++++++++++++++++++++++ scripts/prepare-bundled-package.mjs | 44 ++++++++++++- scripts/release-lib.test.mjs | 54 +++++++++++++-- 3 files changed, 187 insertions(+), 6 deletions(-) diff --git a/scripts/acpx-patch-packaging.test.mjs b/scripts/acpx-patch-packaging.test.mjs index 8aa45833af..d5ef5b817d 100644 --- a/scripts/acpx-patch-packaging.test.mjs +++ b/scripts/acpx-patch-packaging.test.mjs @@ -1,5 +1,17 @@ import assert from "node:assert/strict"; +import { execFileSync } from "node:child_process"; +import { + existsSync, + lstatSync, + mkdirSync, + mkdtempSync, + readFileSync, + rmSync, + writeFileSync, +} from "node:fs"; import { readFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; import test from "node:test"; import cliEsbuildConfig from "../cli/esbuild.config.mjs"; @@ -33,6 +45,87 @@ test("bundled package staging materializes publishConfig entrypoints", () => { assert.deepEqual(staged.exports, adapterUtilsPackage.publishConfig.exports); }); +test("bundled package staging rebuilds npm dependencies and applies the acpx patch", (t) => { + const fixtureDir = mkdtempSync(join(tmpdir(), "paperclip-bundled-stage-")); + const sourceDir = join(fixtureDir, "source"); + const destinationDir = join(fixtureDir, "destination"); + const binDir = join(fixtureDir, "bin"); + const callLog = join(fixtureDir, "calls.log"); + mkdirSync(sourceDir); + mkdirSync(destinationDir); + mkdirSync(binDir); + writeFileSync(join(sourceDir, "package.json"), JSON.stringify(adapterUtilsPackage)); + writeFileSync(callLog, ""); + t.after(() => rmSync(fixtureDir, { recursive: true, force: true })); + + const writeExecutable = (name, body) => { + writeFileSync(join(binDir, name), body, { mode: 0o755 }); + }; + writeExecutable( + "pnpm", + `#!/usr/bin/env bash +set -euo pipefail +printf 'pnpm %s\\n' "$*" >> "$FAKE_CALL_LOG" +destination="\${!#}" +cp "$FAKE_SOURCE_PACKAGE" "$destination/package.json" +mkdir -p "$destination/node_modules/.pnpm" +`, + ); + writeExecutable( + "npm", + `#!/usr/bin/env bash +set -euo pipefail +printf 'npm %s\\n' "$*" >> "$FAKE_CALL_LOG" +[ "$*" = "install --omit=dev --ignore-scripts --no-audit --no-fund" ] +mkdir -p node_modules/acpx/dist +printf 'unpatched runtime\\n' > node_modules/acpx/dist/runtime.js +`, + ); + writeExecutable( + "patch", + `#!/usr/bin/env bash +set -euo pipefail +printf 'patch %s\\n' "$*" >> "$FAKE_CALL_LOG" +target="" +while [ "$#" -gt 0 ]; do + if [ "$1" = "-d" ]; then + target="$2" + shift 2 + else + shift + fi +done +patch_input="$(cat)" +grep -q onAgentStderr <<< "$patch_input" +printf 'patched onAgentStderr runtime\\n' > "$target/dist/runtime.js" +`, + ); + + execFileSync( + process.execPath, + [new URL("./prepare-bundled-package.mjs", import.meta.url).pathname, sourceDir, destinationDir], + { + env: { + ...process.env, + PATH: `${binDir}:${process.env.PATH}`, + FAKE_CALL_LOG: callLog, + FAKE_SOURCE_PACKAGE: join(sourceDir, "package.json"), + }, + stdio: "pipe", + }, + ); + + const stagedAcpxDir = join(destinationDir, "node_modules/acpx"); + assert.equal(lstatSync(stagedAcpxDir).isDirectory(), true); + assert.equal(lstatSync(stagedAcpxDir).isSymbolicLink(), false); + assert.equal(existsSync(join(destinationDir, "node_modules/.pnpm")), false); + assert.match(readFileSync(join(stagedAcpxDir, "dist/runtime.js"), "utf8"), /onAgentStderr/); + assert.match( + readFileSync(callLog, "utf8"), + /patch -p1 --forward -d .*node_modules\/acpx/, + ); +}); + test("bundled package dry runs preview without querying published versions", () => { assert.match(releaseScript, /run_bundled_npm_pack pack --pack-destination "\$publish_dir"/); assert.match(releaseLib, /BUNDLED_NPM_PACK_VERSION="10\.9\.7"/); @@ -40,4 +133,6 @@ test("bundled package dry runs preview without querying published versions", () assert.match(releaseLib, /npx --yes "npm@\$BUNDLED_NPM_PACK_VERSION"/); assert.match(releaseLib, /npx --yes "npm@\$BUNDLED_NPM_PUBLISH_VERSION"/); assert.match(releaseLib, /"\$@" --loglevel verbose/); + assert.match(releaseLib, /run_bundled_npm_publish publish --tag "\$dist_tag"/); + assert.doesNotMatch(releaseLib, /run_bundled_npm_publish publish "\.\/\$tarball"/); }); diff --git a/scripts/prepare-bundled-package.mjs b/scripts/prepare-bundled-package.mjs index 47012959b7..27dbf643c3 100644 --- a/scripts/prepare-bundled-package.mjs +++ b/scripts/prepare-bundled-package.mjs @@ -1,7 +1,7 @@ #!/usr/bin/env node import { execFileSync } from "node:child_process"; -import { readFileSync, writeFileSync } from "node:fs"; +import { readFileSync, rmSync, writeFileSync } from "node:fs"; import { resolve } from "node:path"; import { fileURLToPath } from "node:url"; @@ -19,6 +19,31 @@ export function materializePublishManifest(pkg) { return publishManifest; } +function patchedDependencyPackageName(specifier) { + const versionSeparator = specifier.lastIndexOf("@"); + return versionSeparator > 0 ? specifier.slice(0, versionSeparator) : specifier; +} + +export function applyBundledDependencyPatches(destinationDir, bundledDependencies) { + const rootPackage = JSON.parse(readFileSync(resolve(repoRoot, "package.json"), "utf8")); + const patchedDependencies = rootPackage.pnpm?.patchedDependencies ?? {}; + const bundledDependencyNames = new Set(bundledDependencies); + + for (const [specifier, patchPath] of Object.entries(patchedDependencies)) { + const packageName = patchedDependencyPackageName(specifier); + if (!bundledDependencyNames.has(packageName)) continue; + + execFileSync( + "patch", + ["-p1", "--forward", "-d", resolve(destinationDir, "node_modules", packageName)], + { + input: readFileSync(resolve(repoRoot, patchPath)), + stdio: ["pipe", "inherit", "inherit"], + }, + ); + } +} + export function prepareBundledPackage(sourceDir, destinationDir) { const sourcePackagePath = resolve(sourceDir, "package.json"); const sourcePackage = JSON.parse(readFileSync(sourcePackagePath, "utf8")); @@ -40,6 +65,23 @@ export function prepareBundledPackage(sourceDir, destinationDir) { deployedPackagePath, `${JSON.stringify(materializePublishManifest(deployedPackage), null, 2)}\n`, ); + + rmSync(resolve(destinationDir, "node_modules"), { recursive: true, force: true }); + execFileSync( + "npm", + ["install", "--omit=dev", "--ignore-scripts", "--no-audit", "--no-fund"], + { cwd: destinationDir, stdio: "inherit" }, + ); + applyBundledDependencyPatches(destinationDir, bundledDependencies); + + if ( + bundledDependencies.includes("acpx") && + !readFileSync(resolve(destinationDir, "node_modules/acpx/dist/runtime.js"), "utf8").includes( + "onAgentStderr", + ) + ) { + throw new Error("staged acpx runtime is missing the repository patch"); + } } if (process.argv[1] === fileURLToPath(import.meta.url)) { diff --git a/scripts/release-lib.test.mjs b/scripts/release-lib.test.mjs index 15bf289df4..e33974c8c1 100644 --- a/scripts/release-lib.test.mjs +++ b/scripts/release-lib.test.mjs @@ -78,8 +78,35 @@ if [ "$1" = "view" ] && [ "$NPM_VERSION_EXISTS" = "true" ]; then exit 0 fi if [ "$1" = "publish" ]; then - echo "published" - exit 0 + case "$PNPM_MODE" in + success) + echo "published" + exit 0 + ;; + tlog-then-success) + if [ ! -f "$FAKE_STATE_DIR/npm-called" ]; then + touch "$FAKE_STATE_DIR/npm-called" + echo "npm error code TLOG_CREATE_ENTRY_ERROR" + echo "npm error error creating tlog entry - (409) an equivalent entry already exists in the transparency log with UUID abc" + exit 1 + fi + case " $* " in + *" --provenance=false "*) + echo "published without provenance" + exit 0 + ;; + esac + ;; + tlog-always-fails) + echo "npm error code TLOG_CREATE_ENTRY_ERROR" + echo "npm error error creating tlog entry - (409) an equivalent entry already exists in the transparency log with UUID abc" + exit 1 + ;; + non-tlog-failure) + echo "npm error code E500" + exit 1 + ;; + esac fi exit 1 `, @@ -91,7 +118,9 @@ exit 1 set -euo pipefail printf 'npx %s\n' "$*" >> "$FAKE_CALL_LOG" [ "$1" = "--yes" ] && shift -[ "$1" = "npm@11.18.0" ] && shift +case "$1" in + npm@10.9.7|npm@11.18.0) shift ;; +esac exec npm "$@" `, ); @@ -141,7 +170,7 @@ test("publish_package_to_npm returns after a successful pnpm publish", () => { assert.doesNotMatch(result.calls, /--provenance=false/); }); -test("publish_package_to_npm uses trusted-publishing-capable npm for bundled dependencies", () => { +test("publish_package_to_npm uses trusted publishing from the bundled staging directory", () => { const result = runPublishHelper({ pnpmMode: "success", publishTool: "npm" }); assert.equal(result.status, 0); @@ -149,10 +178,25 @@ test("publish_package_to_npm uses trusted-publishing-capable npm for bundled dep result.calls, /^npx --yes npm@11\.18\.0 publish --tag canary --access public --loglevel verbose$/m, ); - assert.match(result.calls, /^npm publish --tag canary --access public --loglevel verbose$/m); + assert.match( + result.calls, + /^npm publish --tag canary --access public --loglevel verbose$/m, + ); + assert.doesNotMatch(result.calls, / pack /); assert.doesNotMatch(result.calls, /^pnpm publish/m); }); +test("publish_package_to_npm retries bundled directory tlog failures without provenance", () => { + const result = runPublishHelper({ pnpmMode: "tlog-then-success", publishTool: "npm" }); + + assert.equal(result.status, 0); + assert.match(result.calls, /^npm view @paperclipai\/example@1\.2\.3 version$/m); + assert.match( + result.calls, + /^npm publish --tag canary --access public --provenance=false --loglevel verbose$/m, + ); +}); + test("publish_package_to_npm retries duplicate tlog failures without provenance", () => { const result = runPublishHelper({ pnpmMode: "tlog-then-success" });