fix(server): retry cloud-tenant auth sync once on a dropped DB connection (#12773)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Managed-cloud deployments authenticate tenant requests through
trusted headers. The middleware syncs the tenant's user, company, and
membership rows on the way through.
> - Pooled Postgres endpoints sometimes close an established connection
under an in-flight query (pooler recycle, compute suspend). The driver
reconnects on the next query, but the statement on the wire fails.
> - In this path a single dropped statement fails the whole request with
a 500. This happened live on 2026-09-03: the idempotent company
bootstrap insert died with `write CONNECTION_CLOSED`.
> - This pull request retries the actor resolution exactly once when the
error chain carries a postgres.js closed-connection code. The sync is
idempotent end to end, so the replay is safe.
> - The benefit is that a routine pooler blip no longer fails an
authenticated request on the entry path.

## Linked Issues or Issue Description

**What happened?**

A cloud tenant request hit the trusted-header authentication middleware
while the pooled Postgres endpoint closed the connection mid-query. The
insert failed with `write CONNECTION_CLOSED <host>:5432` wrapped in a
`Failed query: insert into "companies" …` error, and the request failed.

**Expected behavior**

The driver reconnects on the next query, and every statement in the
tenant sync is idempotent (upserts, on-conflict inserts, deletes; the
write debounce records only after the full sync succeeds). One
in-request retry should absorb the blip and serve the request.
Non-transient failures must keep failing fast.

**Steps to reproduce**

1. Run an authenticated public deployment against a pooled Postgres
endpoint.
2. Have the pooler close the connection while the middleware's tenant
sync insert is on the wire.
3. Before this change the request fails with a 500; after it the retry
serves the request.

**Deployment mode**

Authenticated public (managed cloud), external pooled PostgreSQL.

## What Changed

- `resolveCloudTenantActor` now delegates to the (unchanged) resolution
body through `retryOnTransientDbConnectionError`, which retries exactly
once on a transient closed-connection failure
- `isTransientDbConnectionError` walks the error `cause` chain (drizzle
wraps the driver error) for the postgres.js codes `CONNECTION_CLOSED`,
`CONNECTION_ENDED`, `CONNECTION_DESTROYED`; both helpers are exported
for tests
- New unit test file `cloud-tenant-transient-db-retry.test.ts`:
detection matrix (including a `23505` staying non-transient),
retry-once-then-succeed, no-retry on non-transient,
propagate-on-second-failure

## Verification

- `pnpm vitest run
src/__tests__/cloud-tenant-transient-db-retry.test.ts` — 5 passed
- `pnpm vitest run
src/__tests__/cloud-tenant-company-provisioning.test.ts` — 7 passed
against embedded Postgres, driving the real resolution path through the
new wrapper

## Risks

- Low risk. The retry is bounded to one attempt, gated on three explicit
driver codes, and wraps an operation that is already idempotent by
design. Every other failure propagates unchanged.
- A genuinely down database now fails after two attempts instead of one
— a few milliseconds of added latency on an already-failing request.

## Model Used

Claude Fable 5 (claude-fable-5) via Claude Code, extended thinking with
tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above (none found for connection-retry work in this path)
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (doc
comments; no user-facing docs affected)
- [x] I have considered and documented any risks above

server/src/__tests__/cloud-tenant-transient-db-retry.test.ts

@@ -0,0 +1,68 @@
+import { describe, expect, it } from "vitest";
+
+import {
+  isTransientDbConnectionError,
+  retryOnTransientDbConnectionError,
+} from "../middleware/auth.ts";
+
+/** The shape drizzle produces: a wrapper whose `cause` is the driver error. */
+function driverClosedError(code: string): Error {
+  const driver = Object.assign(new Error(`write ${code} db.example.internal:5432`), { code });
+  return new Error("Failed query: insert into \"companies\" (…)", { cause: driver });
+}
+
+describe("isTransientDbConnectionError", () => {
+  it("detects a closed-connection code anywhere on the cause chain", () => {
+    expect(isTransientDbConnectionError(driverClosedError("CONNECTION_CLOSED"))).toBe(true);
+    expect(isTransientDbConnectionError(driverClosedError("CONNECTION_ENDED"))).toBe(true);
+    expect(isTransientDbConnectionError(driverClosedError("CONNECTION_DESTROYED"))).toBe(true);
+    const bare = Object.assign(new Error("write CONNECTION_CLOSED host:5432"), {
+      code: "CONNECTION_CLOSED",
+    });
+    expect(isTransientDbConnectionError(bare)).toBe(true);
+  });
+
+  it("rejects everything else", () => {
+    expect(isTransientDbConnectionError(new Error("boom"))).toBe(false);
+    const unique = Object.assign(new Error("duplicate key"), { code: "23505" });
+    expect(isTransientDbConnectionError(unique)).toBe(false);
+    expect(isTransientDbConnectionError(new Error("outer", { cause: unique }))).toBe(false);
+    expect(isTransientDbConnectionError("CONNECTION_CLOSED")).toBe(false);
+    expect(isTransientDbConnectionError(undefined)).toBe(false);
+  });
+});
+
+describe("retryOnTransientDbConnectionError", () => {
+  it("retries exactly once after a transient closed connection", async () => {
+    let calls = 0;
+    const result = await retryOnTransientDbConnectionError(async () => {
+      calls += 1;
+      if (calls === 1) throw driverClosedError("CONNECTION_CLOSED");
+      return "ok";
+    });
+    expect(result).toBe("ok");
+    expect(calls).toBe(2);
+  });
+
+  it("propagates a non-transient failure without retrying", async () => {
+    let calls = 0;
+    await expect(
+      retryOnTransientDbConnectionError(async () => {
+        calls += 1;
+        throw new Error("constraint violation");
+      }),
+    ).rejects.toThrow("constraint violation");
+    expect(calls).toBe(1);
+  });
+
+  it("propagates the second failure when the retry also dies", async () => {
+    let calls = 0;
+    await expect(
+      retryOnTransientDbConnectionError(async () => {
+        calls += 1;
+        throw driverClosedError("CONNECTION_CLOSED");
+      }),
+    ).rejects.toThrow("Failed query");
+    expect(calls).toBe(2);
+  });
+});

server/src/middleware/auth.ts

@@ -512,9 +512,62 @@ export function cloudActorHeaderSourceFromHeaders(
   };
 }
 
+/**
+ * postgres.js codes for a connection the server side closed out from under
+ * an in-flight query — a pooled Postgres endpoint recycling or suspending
+ * (observed 2026-09-03 with a managed pooler closing the socket mid-INSERT).
+ * The driver reconnects transparently on the next query; only the statement
+ * that was on the wire is lost.
+ */
+const transientDbConnectionCodes = new Set([
+  "CONNECTION_CLOSED",
+  "CONNECTION_ENDED",
+  "CONNECTION_DESTROYED",
+]);
+
+/**
+ * True when the error chain (drizzle wraps the driver error as `cause`)
+ * carries a postgres.js closed-connection code. Exported for tests.
+ */
+export function isTransientDbConnectionError(error: unknown): boolean {
+  for (let current: unknown = error; current instanceof Error; current = current.cause) {
+    const code = (current as { code?: unknown }).code;
+    if (typeof code === "string" && transientDbConnectionCodes.has(code)) return true;
+  }
+  return false;
+}
+
+/**
+ * Runs `run` and retries it exactly once when it fails on a transient
+ * closed-connection error. Callers must pass an idempotent operation.
+ * Exported for tests.
+ */
+export async function retryOnTransientDbConnectionError<T>(run: () => Promise<T>): Promise<T> {
+  try {
+    return await run();
+  } catch (error) {
+    if (!isTransientDbConnectionError(error)) throw error;
+    return run();
+  }
+}
+
+/**
+ * Trusted-header actor resolution with a single transient-connection retry.
+ * The tenant sync inside is idempotent end to end — every write is an
+ * upsert/on-conflict/delete and the write debounce records only after the
+ * whole sync succeeds — so replaying it after a dropped connection is safe,
+ * and turns a golden-path authentication 500 into a served request.
+ */
 export async function resolveCloudTenantActor(
   db: Db,
   req: CloudActorHeaderSource,
+): Promise<Express.Request["actor"] | null> {
+  return retryOnTransientDbConnectionError(() => resolveCloudTenantActorOnce(db, req));
+}
+
+async function resolveCloudTenantActorOnce(
+  db: Db,
+  req: CloudActorHeaderSource,
 ): Promise<Express.Request["actor"] | null> {
   const expectedToken = process.env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN?.trim();
   if (!expectedToken) return null;