## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server keeps one postgres.js pool (`packages/db/src/client.ts`,
`createDb`) for every query it runs. #10795 made the pool tunable from
the environment, but the defaults stayed at the driver defaults: an idle
connection never closes, the pool reports itself as `postgres.js`, and
no code path ever calls `sql.end()`.
> - On a hosted Paperclip deployment the server entered a restart loop
(a bundled plugin failure that #12953 describes made every run fail, and
the pool saturated). Each generation opened its ten connections, died,
and left the backends open on the PostgreSQL side until TCP keepalive
reaped them hours later. After about 20 generations the backends
exceeded `max_connections`, and every later boot died on its first
bootstrap query with `sorry, too many clients already`, before
`server.listen()`. The loop could not heal itself. #9555 describes the
same shape on a launchd-supervised self-hosted install.
> - Three properties of the pool combine to make this possible: idle
connections are never reaped, the pool is never ended on any exit path,
and an operator cannot even find the leaked backends in
`pg_stat_activity` because they carry the generic driver name.
> - This pull request gives the pool a 60 second idle timeout and the
`paperclip` application name by default, exposes `max_lifetime` and
`application_name` through the same `DATABASE_*` environment contract
that #10795 introduced, and ends the pool on the orderly SIGINT/SIGTERM
path and on the fail-loud startup path.
> - The benefit is that a restarting or crash-looping server releases
its backends instead of accumulating them, and an operator can see and
count Paperclip's connections.
## Linked Issues or Issue Description
- Refs #9555 — database connection pool leak causes an infinite restart
loop under load. This PR closes the "pool never ends, idle connections
never close" part of that report.
- Refs #12953 — hosted outage report. The pool exhaustion is the second
half of that incident; the first half (a stuck sandbox provider plugin)
has its own PR.
- Related prior PRs: #9597 and #8780 both propose hard-coded
`idle_timeout` / `max_lifetime` values in `createDb`. Both predate
#10795 (merged), which made these options environment-driven; this PR
builds on the merged shape and adds the shutdown `end()` that neither
covers. #4006 and #7481 are closed earlier attempts in the same area.
## What Changed
- `packages/db/src/client.ts`
- New `resolveDatabaseClientOptions()` applies Paperclip defaults on top
of the environment: `idleTimeoutSeconds` defaults to 60
(`DEFAULT_DATABASE_IDLE_TIMEOUT_SECONDS`) and `applicationName` to
`paperclip` (`DEFAULT_DATABASE_APPLICATION_NAME`). `createDb` uses it
for both the environment path and explicit options.
- `DATABASE_IDLE_TIMEOUT_SECONDS` now accepts `0` to restore the driver
default (keep idle connections open). Negative or non-integer values
still throw.
- New environment variables: `DATABASE_MAX_LIFETIME_SECONDS` (positive
integer, maps to `max_lifetime`) and `DATABASE_APPLICATION_NAME`
(non-empty string, maps to `connection.application_name`).
- `postgresJsOptions()` maps the two new options.
- `server/src/shutdown.ts`
- `finalizeServerShutdown` gains two optional ordered steps:
`closeHttpListener` runs first, before the application services stop;
`closeDatabase` runs after the application services and before the
embedded PostgreSQL stop. A failure in either is logged and does not
stop the teardown. Final order: listener → application services →
database pool → embedded PostgreSQL → instrumentation → Sentry.
- New `closeHttpListenerForShutdown()`: stops accepting requests, closes
idle keep-alive sockets, waits up to 5 s for open connections, then
closes whatever is left. Requests still in flight are drained while
every service is available, and none can reach a route after
`sql.end()`, on the signal path and the programmatic path alike (the
programmatic path's later `server.close` finds the listener closed and
skips).
- `server/src/app.ts`: the app shutdown hook (`shutdownAppServices`) now
stops the plugin job scheduler, whose tick queries the database, so a
programmatic `shutdown()` leaves no timer running against the ended
pool.
- `server/src/index.ts`
- `startServer()` is now a thin wrapper around the boot sequence. When
the boot sequence throws after the pool exists, the wrapper ends the
pool (and the separate migration pool, when configured) before it
rethrows. This covers the `process.exit(1)` path in the main module and
the CLI `paperclip run` path alike.
- The orderly shutdown passes the same `closeDatabaseClients` to
`finalizeServerShutdown`.
- `endDatabaseClient` tolerates a client without `$client` (test
doubles) and uses a 5 second end timeout.
- Docs: `docs/deploy/database.md` gets a "Connection Pool Settings"
table with every `DATABASE_*` pool variable, its default and its effect;
`doc/DATABASE.md` lists the two new variables.
- Tests
- `packages/db/src/client-options.test.ts`: parsing of the new
variables, `0` for the idle timeout, rejection of malformed values,
driver option mapping, and the `resolveDatabaseClientOptions` defaults.
- `packages/db/src/client.test.ts` (embedded PostgreSQL):
`createDb(url)` reports `application_name = paperclip` for its own
backend, and a pool with `idleTimeoutSeconds: 1` has zero backends in
`pg_stat_activity` after the timeout.
- `server/src/shutdown.test.ts`: the listener closes before the
application services, and the database close runs between the
application services and the embedded PostgreSQL stop; a failing
database close is logged while the teardown still finishes;
`closeHttpListenerForShutdown` closes idle sockets and resolves on
close, force-closes after the grace period, and is a no-op when the
listener was never bound.
## Verification
- `pnpm --filter @paperclipai/db typecheck` — passes (`check:migrations`
+ `tsc --noEmit`).
- `cd server && pnpm typecheck` — passes.
- `cd packages/db && pnpm exec vitest run src/client-options.test.ts
src/client.test.ts src/client-teardown-registry.test.ts` — 9 + 18 + 3
tests pass (the `client.test.ts` cases need embedded PostgreSQL; the new
one waits up to 10 s for the idle reap and passed in about 3 s).
- `cd server && pnpm exec vitest run src/shutdown.test.ts
src/__tests__/server-startup-feedback-export.test.ts
src/__tests__/bootstrap-claim-routes.test.ts` — 34 + 11 tests pass. The
startup-feedback suite exercises `startServer()` with a mocked
`createDb`, which is why `endDatabaseClient` tolerates a client without
`$client`.
- Manual check for a reviewer: start the server against any PostgreSQL,
then run `SELECT application_name, state, count(*) FROM pg_stat_activity
GROUP BY 1, 2;`. Paperclip's backends now show `paperclip`. Leave the
server idle for more than 60 s and the idle backends disappear. Send
SIGTERM and the backends close before the process exits.
## Risks
- Behavior change with no environment set: idle pooled connections now
close after 60 s. The next query after an idle period pays a reconnect
(single-digit milliseconds on a local socket). postgres.js reconnects
transparently. Set `DATABASE_IDLE_TIMEOUT_SECONDS=0` to keep the
previous behavior.
- `application_name` changes from `postgres.js` to `paperclip`. Anything
that filtered `pg_stat_activity` on the old name would need an update;
nothing in this repo does.
- The HTTP listener now closes at the start of the final teardown (after
the heartbeat run drain, which still needs the API for running agents).
The pool close runs after the application services. A late query from a
timer that survived the service shutdown would fail with a driver
"connection ended" error instead of running; the known database-backed
timer (the plugin job scheduler) is now stopped in the service shutdown.
- The listener drain adds at most 5 s to a shutdown while long-lived
connections (for example WebSocket clients) are open; after that they
are closed forcibly.
- `startServer()` is split into a wrapper and the boot sequence. The
exported signature and return type are unchanged.
- No migration, no schema change.
## Model Used
- Claude Fable 5.1 (`claude-fable-5-1`) via Claude Code, extended
thinking, tool use (file edits, shell, test runs). The change was
produced with the model and reviewed by the submitting human.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_014t3bi2beVNVVHAxK36dmXm
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The Paperclip Runner keeps durable run and provider state outside
one server process.
> - A server restart can leave that runner alive or can interrupt it
after a provider checkpoint.
> - The old startup path used handoff intent and PID evidence, but it
did not reconstruct native ownership.
> - That gap could block the issue, create a replacement run, or start
duplicate provider work.
> - This pull request adds durable same-run recovery for coordinated and
uncoordinated restarts.
> - The benefit is exact recovery of the run, runner, session, provider,
steering, and finalization state.
## Linked Issues or Issue Description
Refs #9628. That pull request added earlier local-adapter hot-restart
work. This change adds native PRP authority reconstruction and same-run
provider resume.
Refs #10935. That pull request handles missing hot-restart snapshots.
This change also supports hard restarts with no snapshot.
Refs #11624. That pull request prevents unsafe retry after an adopted
legacy process exits. This change reconciles native terminal evidence
before provider recovery.
Refs #12070. That pull request improves process liveness checks. This
change also binds recovery to a process-start fingerprint and fails
closed on ambiguity.
**What happened?**
The server could record hot-restart intent, but startup did not rebuild
native runner ownership. A live runner could not re-register its PRP
authority. A dead runner could not resume the exact native and provider
session on the same heartbeat run. Generic recovery could then block the
issue or create replacement work.
**Expected behavior**
A live native runner must reconnect with the same PID and logical
identities. A dead runner must resume the same durable session and
heartbeat run with only a new operating-system PID. A proposed or
terminal result must finalize once before any provider turn starts.
Ambiguous process or session evidence must stay blocked without a signal
or duplicate spawn.
**Steps to reproduce**
1. Start a Paperclip Runner heartbeat and wait for an active provider
turn.
2. Restart only the Paperclip server, with or without a hot-restart
marker.
3. Observe that the old startup path does not reconstruct the native
control-plane authority.
4. Kill both the server and runner after a provider checkpoint.
5. Observe that the old path cannot resume the exact native session on
the original heartbeat run.
**Paperclip version or commit**
The defect was reproduced from commit
`1991f31fd53e7f7794d5c2e4b93be384ade2b41d`. This branch is rebased onto
the current `master`.
**Deployment mode**
Local development and self-hosted server deployments that use the local
Paperclip Runner.
## What Changed
- Added correlated hot-restart requests and version-compatible native
handoff fields.
- Added controller boot identity, process-start identity, controller
generation, recovery state, request id, and bounded history to the
native finalization ledger.
- Added transactional recovery claims for live-runner reattach,
dead-runner resume, and incomplete bootstrap.
- Added fail-closed ownership takeover rules and process identity
validation.
- Added live runner adoption to the local runner transport without a
duplicate spawn.
- Added same-run provider checkpoint resume and legacy retry-row
compatibility.
- Reconciled proposed and terminal results before runner or provider
recovery.
- Bound the HTTP and PRP listener before startup recovery and delayed
scheduling and generic reapers until classification completes.
- Added restart-aware health diagnostics, run-log recovery transitions,
durable runner diagnostics, and bounded shutdown finalizer draining.
- Moved restart-survivable diagnostics into runner-owned, pre-redacted
bounded writes; raw stdout and stderr are never persisted.
- Added process-start fencing for controller, runner, and provider PIDs;
startup classifies every candidate without an implicit cap.
- Added crash-recoverable, contention-safe development restart-request
coordination and failed-startup listener cleanup.
- Added a credential-free real-process restart suite for eight restart,
scale, and identity scenarios.
- Documented native restart operation, persistence, diagnostics, and
verification.
## Verification
- The documented native restart commands passed. They ran eight
real-process/database recovery scenarios and the live runner adoption
transport test.
- Native executor tests passed: 111 tests.
- Heartbeat recovery tests passed: 124 tests.
- Hot restart, health, and shutdown tests passed: 52 tests.
- The broader affected server suite passed: 350 tests.
- Focused native recovery and startup tests passed: 49 tests.
- Runner transport and control-plane tests passed: 63 tests.
- Runner-owned diagnostic tests passed for write-time bounding,
credential redaction, private file modes, and raw stream
non-persistence.
- Development restart coordination tests passed: 11 tests.
- Database migration checks and the partial-application/replay
regression test passed.
- Server, database, and Paperclip Runner typechecks passed.
- `git diff --check` passed.
- Full Paperclip PR CI passed, including build, canary, all five general
server shards, all five serialized server shards, all three browser E2E
shards, workspace suites, and release-registry verification.
- Greptile completed at 5/5 with no outstanding findings,
recommendations, follow-ups, or open review threads.
## Risks
- Moderate risk. This changes startup ordering and ownership transfer
for active native runs.
- The migration adds nullable columns and does not rewrite existing
rows.
- Recovery fails closed when process or durable session identity is
incomplete or contradictory.
- The first implementation supports the local Paperclip Runner. Remote
targets keep their existing behavior.
- The real-process suite covers cleanup and asserts that no runner or
provider process survives each test.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- OpenAI Codex with GPT-5. The runtime did not expose a more specific
model revision or context-window size. Repository editing, shell
execution, database tests, and real-process test execution were enabled.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server and the browser need clear error reports when an operator
enables external monitoring.
> - Paperclip already uses an opt-in OpenTelemetry pattern for server
traces.
> - Sentry can provide error reports for both runtime paths when the
operator sets one data source name.
> - This pull request adds one opt-in Sentry gate for the server and the
browser.
> - The benefit is faster diagnosis while the default setup sends no
Sentry data.
## Linked Issues or Issue Description
**What is improved?**
Paperclip gains optional error monitoring for server and browser
failures.
**Subsystem affected**
Cross-cutting (server, UI, and shared authentication data).
**Current behavior**
Paperclip has no built-in Sentry error capture for server failures or
browser boundary failures. Operators must inspect local logs and browser
tools.
**Proposed behavior**
When the operator sets `SENTRY_DSN`, the server and authenticated
browser use the same Sentry project. When the variable is absent, both
paths stay inactive. The server loads Sentry dynamically and fails open
when the optional package is absent.
**Reason and benefit**
Operators can inspect runtime errors in one Sentry project. The default
setup remains local and sends no monitoring data.
**Breaking changes**
None when `SENTRY_DSN` remains unset. Authenticated session responses
add the optional `sentryDsn` field.
**Additional context**
The implementation uses built-in Sentry privacy options. It disables
default HTTP context and breadcrumb integrations and keeps
`sendDefaultPii` false.
## What Changed
- Add an opt-in server Sentry gate with dynamic package loading and
fail-open behavior.
- Add the Sentry data source name to the authenticated session response.
- Add an authenticated browser Sentry gate and React error boundary
capture.
- Add tests for server, browser, route, and application error paths.
- Document activation, installation, privacy settings, capture behavior,
and operator controls.
## Verification
- Run `npx vitest run server/src/__tests__/sentry.test.ts`.
- Run `npx vitest run ui/src/lib/sentry.test.ts`.
- Run `npx vitest run server/src/__tests__/auth-routes.test.ts
server/src/__tests__/shutdown.test.ts`.
- Confirm that the full continuous integration suite passes on this pull
request.
- Leave `SENTRY_DSN` unset and confirm that the server and browser gates
stay inactive.
- Set `SENTRY_DSN` and install the optional Sentry packages before a
manual capture check.
## Risks
The operator controls the Sentry project and accepts the data risk when
the operator enables the feature. Error objects can contain messages,
stacks, or cause chains with private values. The default configuration
sends no data because the feature stays off without `SENTRY_DSN`. A
missing optional server package does not stop server boot.
## Model Used
OpenAI Codex, GPT-5, with tool use, repository inspection, GitHub CLI
operations, and code review support.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The Claude local adapter supports subscription login through a
sandbox
> - The new-agent page must show login before the user creates an agent
> - Test results must not expose raw sandbox diagnostics or secret
values
> - This pull request adds the login UI to both Test lanes and closes
the diagnostic boundary
> - The branch also adds durable cleanup recovery for failed sandbox
teardown
> - Reusable sandboxes must retain both their recorded teardown
configuration and a valid lifecycle path until destruction succeeds
> - The benefit is a usable login flow with fixed public checks,
redacted server logs, and recoverable sandbox cleanup
## Linked Issues or Issue Description
Related public work:
[#9488](https://github.com/paperclipai/paperclip/pull/9488) adds
first-class recognition for `CLAUDE_CODE_OAUTH_TOKEN` in headless and
remote runs. Related public issue:
[#2681](https://github.com/paperclipai/paperclip/issues/2681) requests
Claude Code subscription support. This pull request adds the login
transport and new-agent UI flow that those changes do not provide.
**Subsystem affected:** Claude local adapter, server login probes,
sandbox provider setup, cleanup recovery, and the new-agent UI.
**Problem or motivation:** The Test lanes did not show the sandbox login
panel in all supported cases. Test results also exposed raw probe
diagnostics, and JSON escapes could end secret redaction early.
**Proposed solution:** Surface the login capability through the bundled
provider manifest. Prepare the same probe runtime in the ACP lane. Send
diagnostics only to redacted server logs. Keep Test checks on fixed
public messages. Normalize login URL hints to allowlisted HTTPS Claude
and Anthropic hosts. Consume JSON escapes during redaction. Preserve
failed sandbox cleanup state across retries and restarts, and prevent
deletion from severing the lifecycle context of a live reusable sandbox.
**Alternatives considered:** Keep raw diagnostics in Test checks or
trust login URL text from the sandbox. Both choices increase information
exposure. Keep separate probe behavior in the ACP lane. That choice
would leave the two Test lanes inconsistent.
## What Changed
- Surface the sandbox login panel on both Test lanes.
- Reconcile the bundled Daytona plugin manifest so
`supportsSetupTokenLogin` reaches the UI capability gate.
- Prepare the ACP Test lane with the same probe runtime as the CLI Test
lane.
- Add the `claude_acp_login_probe_unavailable` warning when the ACP
probe cannot run.
- Send raw sandbox diagnostics only to redacted server logs.
- Keep Test checks on fixed public messages in the ACP, managed-config,
and CLI paths.
- Normalize login URL hints to allowlisted HTTPS Claude and Anthropic
hosts.
- Redact JSON and escaped-JSON secret values, including escaped quotes
and backslashes.
- Preserve orphan cleanup records across provider failures, restarts,
and unavailable plugins.
- Atomically block environment deletion while a live reusable sandbox
lease still depends on it.
- Verify pending cleanup destroys plugin sandboxes with the provider
configuration recorded on the lease, even after the current environment
configuration changes.
## Verification
- Head under review: `506b7fa2d83c36bfa5fd722ee9d95b0c7431c241`.
- Focused environment route/service/runtime coverage passes: 196 tests
across 3 files.
- `pnpm -r typecheck` passes.
- `pnpm build` passes.
- The full Vitest run completed with 4,754 passing and 28 failing tests.
All 23 source-test failures reproduce unchanged on parent head
`58cfe61a33191ce03d965d65085d26064b4888ba`; the other 5 are duplicate
executions from stale `server/dist` output. The failures are unrelated
macOS path/listener and scheduler-fixture failures, so there is no new
bad commit for bisect to localize.
- All required CI checks pass for the current head, including build,
typecheck/release registry, all server and workspace shards, serialized
server suites, canary, and e2e.
- A fresh Greptile review for `506b7fa2d83c36bfa5fd722ee9d95b0c7431c241`
reports 5/5, “safe to merge,” with no blocking failure remaining.
## Risks
- A probe or redaction change could hide useful server diagnostics.
- An allowlist change could reject a valid Claude login URL.
- Cleanup recovery changes could affect provider teardown ordering.
- An environment with a live reusable sandbox can no longer be deleted
until the owning issue or execution workspace completes teardown.
- The implementation keeps public Test messages fixed and sends detail
to redacted server logs.
## Model Used
OpenAI GPT-5 via Codex — exact model ID: GPT-5; tool use and code
execution enabled; extended reasoning enabled. The implementation author
used AI-assisted development.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and documented the result
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation or confirmed no separate
documentation change is needed
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - A guarded hot restart must preserve or finalize every active agent
run.
> - The server uses embedded PostgreSQL when `DATABASE_URL` is not set.
> - The database dependency installs signal handlers before Paperclip
installs its coordinated shutdown handler.
> - Those handlers can stop PostgreSQL before Paperclip writes the
shutdown snapshot.
> - ACP runs also use server-owned stdio and cannot be adopted after
that server exits.
> - This pull request keeps PostgreSQL available through snapshot and
drain, then uses the existing ordered stop.
> - The benefit is a complete restart report with no false adoption and
no missing snapshot loss.
## Linked Issues or Issue Description
**What happened?**
A guarded hot restart with a valid marker can report a live preflight
run as lost with reason `missing_shutdown_snapshot`. The
`embedded-postgres` package imports `async-exit-hook`. That package
registers `SIGINT` and `SIGTERM` listeners before Paperclip registers
its own shutdown listener. The dependency can close PostgreSQL while
Paperclip queries active heartbeat runs and writes the snapshot.
**Expected behavior**
Paperclip must keep its database available until it persists the
shutdown snapshot and completes any required run drain. A detached CLI
run must remain eligible for adoption. An ACP run must finish as
interrupted and queue a retry because its server-owned stdio cannot
survive the server.
**Steps to reproduce**
1. Run Paperclip from source with embedded PostgreSQL.
2. Start a local ACP-backed agent run.
3. Write a valid hot-restart marker for the current server process.
4. send `SIGTERM` through the service manager.
5. Inspect the restart report and server log.
6. Observe that PostgreSQL can close before the shutdown snapshot query
completes.
**Paperclip version or commit**
The defect reproduces on `2ab797dcbed0031c45c7335a0f497fea2a20bd9a`.
**Deployment mode**
Self-hosted server built from source, with embedded PostgreSQL and a
systemd service.
Related work: #9628 introduced hot-restart continuity. #10556 explores a
broader database ownership transfer. #10775 addresses ACP continuity
after replacement startup. This pull request uses a smaller path: it
keeps the current database owner alive through snapshot and drain, then
performs the existing explicit database stop.
## What Changed
- Remove only the `SIGINT` and `SIGTERM` listeners added by the embedded
PostgreSQL import.
- Preserve Paperclip's existing ordered database stop after heartbeat
snapshot and drain.
- Detect active ACP and server-stdio local runs before shutdown.
- Persist their complete snapshot before changing the marker to an ACP
drain request.
- Drain only ACP runs to an interrupted terminal state and queue their
retry.
- Keep detached CLI runs eligible for adoption in the same mixed
restart.
- Quiesce already-running scheduler queue claims before capturing the
snapshot and selective drain set.
- Report a selected ACP run as lost if process termination succeeds but
its terminal database write does not persist.
- Add the drain reason to the restart report.
- Document the normal path and the one-time recovery path across an
older affected build.
## Verification
- `PAPERCLIP_TEST_DATABASE_MODE=native pnpm --filter @paperclipai/server
exec vitest run src/__tests__/heartbeat-process-recovery.test.ts` — 100
passed.
- `pnpm exec vitest run server/src/shutdown.test.ts
server/src/services/hot-restart.test.ts` — 24 passed.
- The shutdown suite imports the real `embedded-postgres` package and
verifies that its eager signal listeners are absent after the guarded
import.
- The embedded PostgreSQL recovery suite verifies snapshot, pre-snapshot
scheduler quiescence, selective ACP drain, detached CLI adoption, queued
retry, original-run finalization, `lostRunIds=[]` in a mixed restart,
and fail-closed reporting when terminal persistence fails.
- `pnpm --filter @paperclipai/server typecheck` — passed.
- `git diff --check` — passed.
- A full workspace typecheck reached the UI and stopped because the
shared local install does not contain its declared `@base-ui/react`
dependency. All server and preceding package checks passed. CI uses a
clean install and remains the authoritative full gate.
## Risks
- Low to moderate risk. This changes shutdown signal ownership and local
run behavior during guarded restarts.
- Paperclip already stops its managed embedded database explicitly. The
change removes only the dependency listeners that race the coordinated
path.
- ACP runs now retry instead of receiving an unsafe bare-process
adoption. Detached CLI runs keep their existing adoption behavior.
- The report adds one field. There is no schema migration or breaking
API change.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- OpenAI Codex with GPT-5. The runtime did not expose a more specific
model revision or context-window size. Reasoning, repository editing,
shell execution, and test execution were enabled.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source control plane people use to manage AI
agents and their work
> - The server coordinates agent heartbeats and preserves eligible live
runs during a hot restart
> - Shutdown previously waited for all heartbeat scheduler work before
capturing the hot-restart snapshot
> - A deployment heartbeat can itself be in that scheduler set while
waiting for the restart, creating a circular wait
> - The missing snapshot prevents startup from classifying and adopting
the still-running agent process
> - This pull request captures the snapshot first and skips
scheduler/drain waits only for an eligible hot restart
> - The benefit is a single SIGTERM can restart the server without
losing eligible live agent runs
## Linked Issues or Issue Description
- **Preflight:** Searched open and closed PRs for the hot-restart
shutdown deadlock; no duplicate found. Reproduced on `master` and
confirmed this is core Paperclip behavior.
- **What happened:** During a hot restart initiated by a running
deployment heartbeat, the SIGTERM handler waited for
`heartbeatSchedulerInFlight` before calling
`prepareHotRestartShutdown()`. The heartbeat was itself in that set and
waited for restart completion, so shutdown never wrote the adoption
snapshot.
- **Expected behavior:** An eligible hot restart captures its snapshot
before waiting for scheduler work, preserves live child processes, and
exits after one SIGTERM.
- **Steps to reproduce:**
1. Start a heartbeat that remains active while requesting a hot restart.
2. Send SIGTERM to the server process.
3. Observe shutdown waiting on the active scheduler task and startup
finding an intent without a shutdown snapshot.
- **Paperclip commit:** `992389480a243b97bda214227e0767eb8c3672af`
- **Deployment/install:** Self-hosted server built from source.
- **Adapter:** Not adapter-specific; reproduced with a Codex heartbeat.
- **Database/access:** Embedded PGlite; agent bearer context.
- **Environment:** Node `v22.22.2` on `Linux 6.17.0-1015-aws aarch64
GNU/Linux`.
- **Privacy:** No secrets, private logs, user paths, or internal issue
references are included.
## What Changed
- Add a focused shutdown coordinator that prepares hot-restart state
before waiting for heartbeat scheduler idleness.
- Skip scheduler-idle and graceful-drain waits only when the hot-restart
service returns `skipDrain: true`.
- Preserve normal graceful shutdown behavior when no eligible intent
exists or preparation fails.
- Add regression coverage for pending scheduler work, normal shutdown,
and preparation failure.
## Verification
- `pnpm exec vitest run server/src/shutdown.test.ts` — 3 passed.
- `pnpm --filter @paperclipai/server typecheck` — passed.
- `pnpm exec vitest run
server/src/__tests__/heartbeat-process-recovery.test.ts -t
'hot-restart'` — 3 passed, 88 skipped.
- `git diff --check origin/master...HEAD` — passed.
## Risks
- Low-to-moderate risk: shutdown ordering changes, but only the
explicitly eligible hot-restart path bypasses scheduler-idle and
run-drain waits.
- Normal shutdown and hot-restart preparation failures retain the
existing graceful behavior.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. This is a focused bug fix
and does not duplicate planned roadmap work.
## Model Used
- OpenAI Codex coding agent; exact runtime model ID and context-window
size are not exposed to the agent. Tool use, shell execution, repository
editing, and test execution were enabled.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (no
documentation change required for this internal shutdown-order fix)
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
Jannes Stubbemann
Dotta
Nicky Leach