From 2177b85eb5116b11bdcfb8376e81dbd6dcd2645b Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Thu, 3 Sep 2026 12:16:55 -0700 Subject: [PATCH] fix(server): retry cloud-tenant auth sync once on a dropped DB connection (#12773) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Managed-cloud deployments authenticate tenant requests through trusted headers. The middleware syncs the tenant's user, company, and membership rows on the way through. > - Pooled Postgres endpoints sometimes close an established connection under an in-flight query (pooler recycle, compute suspend). The driver reconnects on the next query, but the statement on the wire fails. > - In this path a single dropped statement fails the whole request with a 500. This happened live on 2026-09-03: the idempotent company bootstrap insert died with `write CONNECTION_CLOSED`. > - This pull request retries the actor resolution exactly once when the error chain carries a postgres.js closed-connection code. The sync is idempotent end to end, so the replay is safe. > - The benefit is that a routine pooler blip no longer fails an authenticated request on the entry path. ## Linked Issues or Issue Description **What happened?** A cloud tenant request hit the trusted-header authentication middleware while the pooled Postgres endpoint closed the connection mid-query. The insert failed with `write CONNECTION_CLOSED :5432` wrapped in a `Failed query: insert into "companies" …` error, and the request failed. **Expected behavior** The driver reconnects on the next query, and every statement in the tenant sync is idempotent (upserts, on-conflict inserts, deletes; the write debounce records only after the full sync succeeds). One in-request retry should absorb the blip and serve the request. Non-transient failures must keep failing fast. **Steps to reproduce** 1. Run an authenticated public deployment against a pooled Postgres endpoint. 2. Have the pooler close the connection while the middleware's tenant sync insert is on the wire. 3. Before this change the request fails with a 500; after it the retry serves the request. **Deployment mode** Authenticated public (managed cloud), external pooled PostgreSQL. ## What Changed - `resolveCloudTenantActor` now delegates to the (unchanged) resolution body through `retryOnTransientDbConnectionError`, which retries exactly once on a transient closed-connection failure - `isTransientDbConnectionError` walks the error `cause` chain (drizzle wraps the driver error) for the postgres.js codes `CONNECTION_CLOSED`, `CONNECTION_ENDED`, `CONNECTION_DESTROYED`; both helpers are exported for tests - New unit test file `cloud-tenant-transient-db-retry.test.ts`: detection matrix (including a `23505` staying non-transient), retry-once-then-succeed, no-retry on non-transient, propagate-on-second-failure ## Verification - `pnpm vitest run src/__tests__/cloud-tenant-transient-db-retry.test.ts` — 5 passed - `pnpm vitest run src/__tests__/cloud-tenant-company-provisioning.test.ts` — 7 passed against embedded Postgres, driving the real resolution path through the new wrapper ## Risks - Low risk. The retry is bounded to one attempt, gated on three explicit driver codes, and wraps an operation that is already idempotent by design. Every other failure propagates unchanged. - A genuinely down database now fails after two attempts instead of one — a few milliseconds of added latency on an already-failing request. ## Model Used Claude Fable 5 (claude-fable-5) via Claude Code, extended thinking with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above (none found for connection-retry work in this path) - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (doc comments; no user-facing docs affected) - [x] I have considered and documented any risks above --- .../cloud-tenant-transient-db-retry.test.ts | 68 +++++++++++++++++++ server/src/middleware/auth.ts | 53 +++++++++++++++ 2 files changed, 121 insertions(+) create mode 100644 server/src/__tests__/cloud-tenant-transient-db-retry.test.ts diff --git a/server/src/__tests__/cloud-tenant-transient-db-retry.test.ts b/server/src/__tests__/cloud-tenant-transient-db-retry.test.ts new file mode 100644 index 0000000000..70ada7397d --- /dev/null +++ b/server/src/__tests__/cloud-tenant-transient-db-retry.test.ts @@ -0,0 +1,68 @@ +import { describe, expect, it } from "vitest"; + +import { + isTransientDbConnectionError, + retryOnTransientDbConnectionError, +} from "../middleware/auth.ts"; + +/** The shape drizzle produces: a wrapper whose `cause` is the driver error. */ +function driverClosedError(code: string): Error { + const driver = Object.assign(new Error(`write ${code} db.example.internal:5432`), { code }); + return new Error("Failed query: insert into \"companies\" (…)", { cause: driver }); +} + +describe("isTransientDbConnectionError", () => { + it("detects a closed-connection code anywhere on the cause chain", () => { + expect(isTransientDbConnectionError(driverClosedError("CONNECTION_CLOSED"))).toBe(true); + expect(isTransientDbConnectionError(driverClosedError("CONNECTION_ENDED"))).toBe(true); + expect(isTransientDbConnectionError(driverClosedError("CONNECTION_DESTROYED"))).toBe(true); + const bare = Object.assign(new Error("write CONNECTION_CLOSED host:5432"), { + code: "CONNECTION_CLOSED", + }); + expect(isTransientDbConnectionError(bare)).toBe(true); + }); + + it("rejects everything else", () => { + expect(isTransientDbConnectionError(new Error("boom"))).toBe(false); + const unique = Object.assign(new Error("duplicate key"), { code: "23505" }); + expect(isTransientDbConnectionError(unique)).toBe(false); + expect(isTransientDbConnectionError(new Error("outer", { cause: unique }))).toBe(false); + expect(isTransientDbConnectionError("CONNECTION_CLOSED")).toBe(false); + expect(isTransientDbConnectionError(undefined)).toBe(false); + }); +}); + +describe("retryOnTransientDbConnectionError", () => { + it("retries exactly once after a transient closed connection", async () => { + let calls = 0; + const result = await retryOnTransientDbConnectionError(async () => { + calls += 1; + if (calls === 1) throw driverClosedError("CONNECTION_CLOSED"); + return "ok"; + }); + expect(result).toBe("ok"); + expect(calls).toBe(2); + }); + + it("propagates a non-transient failure without retrying", async () => { + let calls = 0; + await expect( + retryOnTransientDbConnectionError(async () => { + calls += 1; + throw new Error("constraint violation"); + }), + ).rejects.toThrow("constraint violation"); + expect(calls).toBe(1); + }); + + it("propagates the second failure when the retry also dies", async () => { + let calls = 0; + await expect( + retryOnTransientDbConnectionError(async () => { + calls += 1; + throw driverClosedError("CONNECTION_CLOSED"); + }), + ).rejects.toThrow("Failed query"); + expect(calls).toBe(2); + }); +}); diff --git a/server/src/middleware/auth.ts b/server/src/middleware/auth.ts index e2e4b84d59..1125cbf4f7 100644 --- a/server/src/middleware/auth.ts +++ b/server/src/middleware/auth.ts @@ -512,9 +512,62 @@ export function cloudActorHeaderSourceFromHeaders( }; } +/** + * postgres.js codes for a connection the server side closed out from under + * an in-flight query — a pooled Postgres endpoint recycling or suspending + * (observed 2026-09-03 with a managed pooler closing the socket mid-INSERT). + * The driver reconnects transparently on the next query; only the statement + * that was on the wire is lost. + */ +const transientDbConnectionCodes = new Set([ + "CONNECTION_CLOSED", + "CONNECTION_ENDED", + "CONNECTION_DESTROYED", +]); + +/** + * True when the error chain (drizzle wraps the driver error as `cause`) + * carries a postgres.js closed-connection code. Exported for tests. + */ +export function isTransientDbConnectionError(error: unknown): boolean { + for (let current: unknown = error; current instanceof Error; current = current.cause) { + const code = (current as { code?: unknown }).code; + if (typeof code === "string" && transientDbConnectionCodes.has(code)) return true; + } + return false; +} + +/** + * Runs `run` and retries it exactly once when it fails on a transient + * closed-connection error. Callers must pass an idempotent operation. + * Exported for tests. + */ +export async function retryOnTransientDbConnectionError(run: () => Promise): Promise { + try { + return await run(); + } catch (error) { + if (!isTransientDbConnectionError(error)) throw error; + return run(); + } +} + +/** + * Trusted-header actor resolution with a single transient-connection retry. + * The tenant sync inside is idempotent end to end — every write is an + * upsert/on-conflict/delete and the write debounce records only after the + * whole sync succeeds — so replaying it after a dropped connection is safe, + * and turns a golden-path authentication 500 into a served request. + */ export async function resolveCloudTenantActor( db: Db, req: CloudActorHeaderSource, +): Promise { + return retryOnTransientDbConnectionError(() => resolveCloudTenantActorOnce(db, req)); +} + +async function resolveCloudTenantActorOnce( + db: Db, + req: CloudActorHeaderSource, ): Promise { const expectedToken = process.env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN?.trim(); if (!expectedToken) return null;