## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The CLI test suite uses embedded Postgres for worktree checks.
> - Several tests start a cluster and run migrations on each test.
> - Runner contention can make this cost class exceed small hand-written
time budgets.
> - A timed-out test can also leave the cluster alive while cleanup
removes its data directory.
> - This pull request gives the cost class one measured timeout and
registers cleanup for the timeout path.
> - The benefit is stable tests and fewer orphaned embedded Postgres
processes.
## Linked Issues or Issue Description
No public GitHub issue tracks this defect. Related PRs #10987 and #10961
address wider embedded-Postgres test setup issues. This pull request
addresses the CLI timeout and cleanup path described below.
**What happened?**
Five CLI tests used different hand-written time budgets while they
started embedded Postgres and ran migrations. Runner contention made one
test exceed its budget. A timeout also skipped the `finally` cleanup
path and left a Postgres process alive.
**Expected behavior**
Each embedded-Postgres test gets a budget that covers measured runner
contention. Cleanup stops the cluster before the test removes its data
directory, including after a timeout.
**Steps to reproduce**
1. Run `npx vitest run cli/src/__tests__/worktree.test.ts` on a
contended runner.
2. Observe the embedded-Postgres tests take longer than their small
hand-written budgets.
3. Inspect the process list after a timeout and observe an orphaned
Postgres process.
**Paperclip version or commit**
The defect reproduces on `master` before this pull request.
**Deployment mode**
Built from source with Vitest.
**Installation method**
Built from source with pnpm.
## What Changed
- Export `EMBEDDED_POSTGRES_TEST_TIMEOUT_MS` from the embedded-Postgres
test helper.
- Apply the shared timeout to every test defined by
`itEmbeddedPostgres`.
- Replace the five hand-written timeout values in `worktree.test.ts`.
- Register temporary-directory removal and cluster stop with
`onTestFinished` in the correct order.
- Restore the working directory during timeout cleanup.
## Verification
- `npx vitest run cli/src/__tests__/worktree.test.ts` passes 63 tests.
- `npx vitest run cli/src` passes 424 tests across 59 files.
- `tsc --noEmit` in `cli/` adds no new error in the touched files.
- CI must pass on the pull request head.
- Greptile must report 5/5 with no unresolved blocking thread.
## Risks
- Low risk. The change affects test helpers and test cleanup only.
- The shared budget can lengthen a failing test before Vitest reports
the failure.
- The cleanup order depends on Vitest callback order, which the tests
now use explicitly.
## Model Used
OpenAI Codex, GPT-5. The model used tool calls and code review support.
The context window size and internal reasoning mode were not exposed in
this run.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip uses database clients and embedded PostgreSQL test
fixtures
> - A fixture stopped its embedded PostgreSQL cluster while clients
still held connections
> - The postgres.js driver then scheduled a write on a stopped
connection
> - That write escaped the timer callback and caused a test process to
exit with an error
> - This pull request closes registered clients before the fixture stops
its cluster
> - The benefit is stable test teardown and clear failure reporting in
continuous integration
## Linked Issues or Issue Description
Refs: #10869
**What happened?**
An embedded PostgreSQL test fixture stopped its cluster while database
clients still held open connections. The postgres.js driver then
scheduled a deferred write on a dead connection. The write caused an
unhandled error after the test shard reported success.
**Expected behavior**
The fixture closes all live clients for its cluster before it stops the
embedded PostgreSQL cluster. Tests then finish without a deferred write
on a dead connection.
**Steps to reproduce**
1. Run the database regression test with the embedded PostgreSQL
fixture.
2. Stop the fixture while its database client still has an open
connection.
3. Observe the deferred write and the process exit status.
**Paperclip version or commit**
Branch base: bdd8f1bed. Change head:
93e85d2ba1.
**Deployment mode**
Local dev with the embedded PostgreSQL test fixture.
**Installation method**
Built from source with pnpm.
**Agent adapter(s) involved**
Not adapter-specific. This change covers database test infrastructure.
**Database mode**
Embedded PGlite.
**Additional context**
The change keeps client references weak and keys them by host and port.
It does not retain credentials. It also handles connection URLs that the
driver accepts when the URL parser rejects them.
## What Changed
- Add a registry for live database clients in the database package.
- Close registered clients before the embedded PostgreSQL fixture stops
its cluster.
- Add a regression test for the teardown race.
- Handle driver-compatible URLs that the standard URL parser rejects.
- Add cleanup for the shared route test harness.
## Verification
- Run the full `packages/db` suite.
- Run `tsc --noEmit` in `packages/db`.
- Run the server suite that uses `route-test-harness.ts`.
- Run the teardown regression test five times.
- Confirm that the negative control fails three times.
- Confirm that no shard reports green tests and exits with an error.
## Risks
The registry changes client cleanup for embedded test fixtures. Weak
references limit retained memory in long-lived processes. The registry
uses host and port only, so it does not retain credentials. No
migration, schema, API, telemetry, authentication, or cryptography
change exists.
## Model Used
OpenAI Codex, GPT-5, tool use and code review support, standard
reasoning mode. The implementing engineer supplied the code and
verification results.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The database layer uses embedded Postgres for isolated test runs
> - A port probe can fail when another process takes the same port
before Postgres binds it
> - That race can make a test fail even when the code under test is fine
> - This pull request adds bounded retry and clearer error text to the
embedded Postgres start path
> - The benefit is more stable tests and faster diagnosis when startup
still fails
## Linked Issues or Issue Description
No public GitHub issue exists for this change.
This PR fixes a flaky embedded Postgres test start path.
The helper can lose a port between probe and bind.
This PR retries the start with a fresh port and a fresh data directory.
Related public context:
- Refs #7259
- Refs #9769
## What Changed
- Add bounded retry around embedded Postgres initialization and start.
- Stop each failed attempt and remove its data directory before the next
attempt.
- Capture Postgres output in the thrown error so the failure is easier
to read.
- Add unit coverage for retry success, retry exhaustion, and the
improved error text.
## Verification
- `pnpm --filter @paperclipai/db exec vitest run
src/test-embedded-postgres.test.ts src/embedded-postgres-error.test.ts`
- `pnpm --filter @paperclipai/db exec vitest run`
- `pnpm --filter @paperclipai/db exec vitest run` passed in the worktree
after the change.
- `worktree.test.ts > quarantines copied live execution state in seeded
worktree databases` passed.
- A real cluster loop of 100 starts passed with 0 failures.
## Risks
- The retry can hide a real startup fault until the fifth try.
- The bound keeps the wait short, and the final error still shows the
captured Postgres log.
- This change only affects the embedded Postgres test start helper.
## Model Used
OpenAI Codex, GPT-5, tool-use enabled.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is an open-source AI agent management platform; its test
suite spans a `server` package that mounts real embedded Postgres
databases in `beforeAll`/`afterAll` hooks
> - The `server` package CI shard runs all ~93 suites serially
(`maxWorkers=1`) on a loaded CI host; each suite boots and tears down
its own embedded Postgres in hook callbacks
> - vitest's default `hookTimeout` is 10 seconds; under load, graceful
embedded-Postgres shutdown occasionally crosses that threshold
> - This produces intermittent `Error: Hook timed out in 10000ms`
failures in `afterAll` hooks — not test assertion failures — and the
suites pass on re-run, making them textbook flaky tests
> - Inspecting `embedded-postgres@18.1.0-beta.16` shows that `stop()`
takes no argument (no fast-shutdown mode), SIGINTs postgres (already
PostgreSQL "fast shutdown"), and resolves only on the child's `exit`
event with no internal time bound
> - Two targeted fixes: (1) raise `hookTimeout` and `teardownTimeout` to
30 s in `server/vitest.config.ts` — one config change that eliminates
the flake for all ~93 suites at once; (2) wrap `stop()` in a 5 s bounded
`Promise.race` in the test helper so a slow shutdown can never hang the
hook regardless of OS scheduling variance
> - This PR changes only test-infra and test-config; no production-code
behavior changes
## Linked Issues or Issue Description
No public GitHub issue exists for this flake. Inline bug description
(bug report template):
**What happened?**
The `General tests (server (N/3))` CI shards intermittently fail with
`Error: Hook timed out in 10000ms` in `afterAll` hooks and pass on
re-run. Every test assertion passes; only the teardown hook exceeds
vitest's default timeout.
**Expected behavior**
CI passes reliably. Teardown timeouts should not be a source of flake.
**Steps to reproduce**
Run the server test suite repeatedly on a loaded host or in CI with
`maxWorkers=1` — the shard occasionally crosses 10 s in `afterAll`
during embedded-Postgres shutdown.
**Paperclip version**
`master`, any build that includes `server/vitest.config.ts` without an
explicit `hookTimeout`.
**Deployment mode**
Self-hosted (CI).
## What Changed
- **`server/vitest.config.ts`** — added `hookTimeout: 30000` and
`teardownTimeout: 30000`. Removes flake across all ~93 server suites at
once. 30 s gives generous headroom over observed worst-case teardown
while still catching a genuinely hung hook.
- **`packages/db/src/test-embedded-postgres.ts`** — added
`stopEmbeddedPostgresBounded()`, a 5 s `Promise.race` wrapper around
`stop()`. Applied at all three call sites inside `cleanup()`. Data dir
is still removed unconditionally; errors are still swallowed; the
null-instance guard is preserved. Existing behavior unchanged except the
shutdown can no longer block indefinitely.
## Verification
- `tsc --noEmit` clean on `packages/db` (built against worktree-local
`shared`)
- `packages/db` `client.test.ts` passes 14/14 — boots embedded Postgres
and exercises the bounded teardown via `cleanup()` in `afterEach`
- Standalone bounded-race semantics verified: hang resolves at the 5 s
bound; late or immediate `stop()` rejection swallowed; no unhandled
rejection; null-instance path safe
- CI: all 3 server shards + split-verify lane (Async-Verification Gate)
expected green after this PR
```bash
# Reproduce the teardown test locally:
cd packages/db && npx vitest run src/client.test.ts
# Type-check packages/db:
npx tsc --noEmit -p packages/db/tsconfig.json
```
## Risks
Low risk. No product-code changes — test-infra and test-config only. The
vitest timeout increase is additive (raises the ceiling; never lowers
it). The bounded race wrapper preserves prior teardown behavior exactly:
data dir always removed, errors always swallowed, stop is still
attempted. A worst-case outcome is that a genuinely hung `stop()` now
surfaces as a test timeout at 30 s instead of 10 s — still caught, just
later.
## Model Used
- **Provider:** Anthropic
- **Model:** Claude Sonnet 4.6 (`claude-sonnet-4-6`)
- **Context window:** 200 k tokens
- **Capabilities:** tool use, code execution, extended reasoning
- **Mode:** Paperclip agent heartbeat (autonomous execution with human
board oversight)
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Harold Kim <harold@paperclip.ing>
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip coordinates autonomous company work through local and
hosted runtime surfaces.
> - Local embedded Postgres and tenant import/export paths are
foundational reliability pieces.
> - A runtime failure in either path can stop agents or imports before
useful work begins.
> - The branch included remaining fixes for embedded native library
bootstrap and async tenant import handling.
> - This pull request groups those runtime/import reliability changes
into one standalone PR.
> - The benefit is a more robust local runtime and safer cloud tenant
import behavior.
## What Changed
- Prepared embedded Postgres native runtime before startup in
CLI/server/test entrypoints.
- Added embedded Postgres native bootstrap coverage.
- Added async tenant import job handling and deferred validation
coverage.
- Kept the runtime/import changes based directly on current
`origin/master` after related upstream PRs had already merged.
## Verification
- `pnpm --filter @paperclipai/plugin-sdk build`
- `NODE_ENV=test pnpm exec vitest run
packages/db/src/embedded-postgres-native.test.ts
server/src/__tests__/company-portability-routes.test.ts`
## Risks
- Medium-low: this touches startup/import paths, but the branch is small
and covered by targeted tests.
- The embedded Postgres change depends on platform-specific
native-library behavior, so CI and follow-up checks should still verify
supported runners.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- OpenAI GPT-5 Codex via `codex_local`, tool-enabled coding session;
exact context window not exposed by this runtime.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] If this change affects the UI, I have included before/after
screenshots
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge
## Thinking Path
> - Paperclip is a control plane for autonomous AI companies, where work
must end with a clear disposition rather than ambiguous agent liveness.
> - Recovery currently detects stalled or missing-next-step issues, but
source issue recovery can become split across child recovery issues,
blockers, and comments.
> - That makes it harder for operators and agents to see who owns
recovery and what exact action is needed on the original issue.
> - Source-scoped recovery actions give the original issue a first-class
active recovery state with owner, evidence, wake policy, and resolution
outcome.
> - This pull request adds the recovery-action data model, backend
reconciliation and resolution APIs, and board UI indicators/actions.
> - The benefit is clearer stalled-work recovery without losing source
issue context or relying on comments as the liveness path.
## What Changed
- Added the `issue_recovery_actions` schema, shared
types/constants/validators, and an idempotent
`0084_issue_recovery_actions` migration ordered after current `master`
migrations.
- Updated stranded/missing-disposition recovery to create source-scoped
recovery actions, wake the recovery owner on the source issue, and avoid
locking the source issue for recovery-action wakes.
- Added API support for reading active recovery actions on issue
detail/list surfaces and resolving them with restored, blocked,
cancelled, or false-positive outcomes.
- Require blocked recovery resolutions to have an unresolved first-class
blocker, and removed the UI shortcut that could mark recovery blocked
without a blocker selection path.
- Surfaced recovery indicators/actions in the issue UI, blocker notices,
active run panels, issue rows, and Storybook coverage.
- Updated docs and focused tests for recovery semantics, ownership,
races, stale comments, and UI behavior.
## Verification
- `pnpm exec vitest run
server/src/__tests__/issue-recovery-actions.test.ts
server/src/__tests__/heartbeat-process-recovery.test.ts
ui/src/components/IssueRecoveryActionCard.test.tsx
ui/src/components/IssueBlockedNotice.test.tsx ui/src/api/issues.test.ts`
— 5 files, 72 tests passed.
- `pnpm --filter @paperclipai/shared typecheck` — passed.
- `pnpm --filter @paperclipai/db typecheck` — passed, including
migration numbering check.
- `pnpm --filter @paperclipai/server typecheck` — passed.
- `pnpm --filter @paperclipai/ui typecheck` — passed.
- Follow-up verification after blocker-resolution guard: `pnpm exec
vitest run server/src/__tests__/issue-recovery-actions.test.ts
ui/src/components/IssueRecoveryActionCard.test.tsx
ui/src/api/issues.test.ts` — 3 files, 27 tests passed.
- Follow-up `pnpm --filter @paperclipai/server typecheck` — passed.
- Follow-up `pnpm --filter @paperclipai/ui typecheck` — passed.
- UI states are available in
`ui/storybook/stories/source-issue-recovery.stories.tsx`; screenshot
capture helper is `scripts/screenshot-recovery-card.cjs`.
## Risks
- Medium: recovery behavior changes from child recovery issue ownership
toward source-scoped actions, so operators may see stalled-work state in
new places.
- Migration risk is mitigated by using the next migration slot after
`master` and making the table/constraints/index creation idempotent for
anyone who previously applied the old branch-local
`0082_dizzy_master_mold` migration.
- Existing child recovery issue paths are still guarded for
already-created recovery issues, but new source-scoped flows should be
watched in CI and Greptile review.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- OpenAI Codex, GPT-5 coding agent, tool use enabled for shell, Git,
GitHub, and local test execution. Context window not exposed by the
runtime.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] If this change affects the UI, I have included before/after
screenshots
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
- probe host support on every platform instead of special-casing darwin
- re-export the db package helper from server and cli tests
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Nicky Leach
Dotta
dotta