## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Managed workspaces run local web services and embedded PostgreSQL
databases
> - A listening process could return an unhealthy response and still be
reused
> - Embedded PostgreSQL failures had no bounded restart owner
> - Cleanup inferred ownership from a branch slug instead of exact
persisted instance data
> - This pull request validates runtime health, supervises database
recovery, and uses exact cleanup ownership
> - The benefit is reliable replacement of degraded services without
deleting active instances
## Linked Issues or Issue Description
**What happened?**
Workspace reconciliation could reuse a degraded Paperclip process after
any successful HTTP response. Embedded PostgreSQL could stop without
bounded recovery. Cleanup could infer database ownership from a branch
slug and select the wrong instance.
**Expected behavior**
Paperclip must require a semantic healthy response from the assigned
loopback listener. It must replace degraded processes. It must supervise
embedded PostgreSQL with bounded restarts. Cleanup must use exact
persisted worktree and instance-root ownership.
**Steps to reproduce**
1. Start a managed workspace runtime.
2. Make its health endpoint return HTTP 200 with an unhealthy status, or
stop its embedded PostgreSQL process.
3. Reconcile the workspace or run instance cleanup.
4. Observe that the old implementation can reuse the degraded runtime or
infer ownership from its branch slug.
**Paperclip version or commit**
Current `master` before this pull request.
**Deployment mode**
Local dev with managed workspace services.
## What Changed
- Require `{ "status": "ok" }` from the assigned loopback health
endpoint before runtime reuse or adoption.
- Refresh persisted runtime health and replace degraded managed
processes.
- Add bounded embedded PostgreSQL restart supervision with coordinated
shutdown and hot-restart support.
- Stop the unhealthy web process when PostgreSQL recovery is exhausted
so reconciliation can replace it.
- Require exact persisted instance-root ownership before cleanup can
reclaim an embedded database.
- Add focused regression tests for degraded HTTP responses, ownership
mismatches, bounded recovery, active instance preservation, and
confirmed orphan reclamation.
## Verification
- `pnpm --filter @paperclipai/server typecheck`
- `pnpm --filter @paperclipai/server test --
src/embedded-postgres-supervisor.test.ts
src/services/workspace-instance-cleanup.test.ts
src/services/workspace-runtime.test.ts
src/services/execution-workspaces-service.test.ts`
- `pnpm -r typecheck`
- `pnpm build`
- `git diff --check`
- The full local stable test runner also found host-owned listeners on
ports 42000 and 52000. Those listeners conflict with the exposure test
fixture. The focused changed suites pass, and CI runs on a clean host.
## Risks
- A custom process that returns HTTP 2xx without the Paperclip health
contract is now degraded by design.
- Restart exhaustion terminates the managed web process. The runtime
reconciler then starts a clean process.
- Cleanup now fails closed when persisted ownership is missing. This can
retain an ambiguous orphan for manual review.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- OpenAI GPT-5 Codex. The exact serving revision and context-window size
are not exposed. The model used agentic reasoning, repository tools,
code execution, and test execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Isolated workspaces give each task a safe and reproducible checkout.
> - The existing setup cloned the development database before an agent
needed to run the app.
> - This made worktree creation slower and heavier for tasks that never
start a service.
> - Runtime services already use one server start path for heartbeat,
operator, and startup recovery flows.
> - This pull request moves heavy setup to that start path and keeps
worktree creation lean.
> - The benefit is faster isolated workspace creation with the same
reliable runtime setup when a service starts.
## Linked Issues or Issue Description
Related pull request: #10652 covers the initial deferred
database-seeding slice. This pull request supersedes it with end-to-end
runtime provisioning and safe cleanup.
**What existing behavior does this improve?**
This improves isolated worktree creation, runtime service startup, and
isolated instance cleanup.
**Subsystem affected**
Cross-cutting: CLI worktree setup, server runtime orchestration, shared
workspace contracts, and development scripts.
**Current behavior**
Paperclip seeds an isolated development database during worktree
creation. It can also leave an isolated instance directory after
workspace teardown. This work happens even when no runtime service
starts.
**Proposed behavior**
Paperclip creates the worktree with a lean eager setup. It runs an
idempotent runtime provision command before the first managed service
spawn. Concurrent starts share one provision attempt. Teardown removes
the isolated instance safely.
**Reason and benefit**
Many agent tasks only edit and test code. They do not need a running
Paperclip instance. Deferring the database seed reduces workspace
startup cost while preserving automatic setup for tasks that start the
app.
**Breaking changes**
None. The new runtime provision command is optional. Existing workspace
behavior is unchanged when it is absent.
## What Changed
- Split Paperclip worktree setup into a lean eager script and an
idempotent runtime provision script.
- Added `runtimeProvisionCommand` to project, issue, realized workspace,
and persisted workspace contracts.
- Added a per-workspace provision mutex before local service spawn for
heartbeat, operator, and startup recovery flows.
- Added a persisted `provisioning` service state and the
`workspace_runtime_provision` operation phase.
- Kept provision time outside the service readiness timeout and made
failed attempts visible and retryable.
- Reclaimed isolated instance data during safe workspace teardown.
- Serialized deferred database seeding across processes and bound
teardown to the instance root captured in persisted workspace metadata.
- Added tests for config flow, concurrency, retry, no-op behavior,
readiness timing, scripts, CLI commands, and cleanup.
- Documented the eager and runtime provisioning contracts.
## Verification
- `pnpm -r typecheck`
- `pnpm build`
- `pnpm test:run` (server: 3,201 passed; UI: 3,345 passed; the CLI phase
exposed one environment-sensitive AWS doctor assertion because the agent
runtime injects static AWS credentials)
- `env -u AWS_ACCESS_KEY_ID -u AWS_SECRET_ACCESS_KEY pnpm exec vitest
run cli/src/__tests__/secrets.test.ts -t 'passes AWS doctor checks when
non-secret provider config is present'`
- Focused runtime tests cover serialized provisioning, retry after
stderr failure, absent-command no-op behavior, operation logging,
persisted state order, and readiness timeout exclusion.
- Focused CLI and cleanup tests cover concurrent seed serialization,
stale-lock fail-closed behavior, persisted instance ownership, and
rewritten sibling pointers.
## Risks
- A faulty runtime provision script blocks service startup. Paperclip
records stderr, marks the service failed, and retries on the next start.
- Concurrent service requests share an in-process provision attempt,
while the seed command uses an atomic filesystem lock across processes.
A stale lock fails closed and requires an operator to verify no seed is
running before removing it.
- Isolated instance cleanup is destructive. The cleanup service
validates ownership and path containment before removal.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- OpenAI Codex, `gpt-5.6-sol`, with agentic reasoning, tool use, and
code execution. The service does not expose the context-window size.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip creates isolated instances for server-managed git
worktrees.
> - The worktree teardown path removes the git worktree but leaves its
isolated instance directory behind.
> - The leaked directory can retain an embedded PostgreSQL process and
database files.
> - Teardown must remove only the collision-resistant instance assigned
to that exact worktree path.
> - This pull request stops the verified embedded PostgreSQL process and
removes the guarded instance directory.
> - The benefit is complete worktree cleanup without risk to another,
default, or live Paperclip instance.
## Linked Issues or Issue Description
**What happened?**
Closing a server-managed git worktree removed the git worktree and
branch, but it left the isolated Paperclip instance directory behind. A
live embedded PostgreSQL process could also keep running against that
directory.
**Expected behavior**
Worktree teardown must stop the isolated embedded PostgreSQL process and
remove only the instance assigned to that exact worktree. It must refuse
mismatched instance IDs and all paths outside
`PAPERCLIP_WORKTREES_DIR/instances/`.
**Steps to reproduce**
1. Create a server-managed git worktree with a repo-local
`.paperclip/.env` file.
2. Start its isolated embedded PostgreSQL instance.
3. Close the execution workspace.
4. Observe that the git worktree is removed but the isolated instance
directory remains.
**Paperclip version or commit**
The bug reproduces on `master` before this change.
**Deployment mode**
Local development with a server-managed git worktree and embedded
PostgreSQL.
## What Changed
- Give server-managed worktrees collision-resistant instance IDs derived
from their resolved absolute paths.
- Capture the repo-local instance pointer before custom teardown
commands can remove it.
- Require the pointer's instance ID to match the exact worktree-derived
ID.
- Resolve and validate the instance path against the canonical managed
worktree instance root.
- Verify and stop the matching embedded PostgreSQL process before
directory removal, including process-exit races.
- Record successful and refused cleanup operations in the workspace
operation log.
- Add focused ownership, process-race, path-safety, and runtime
integration tests.
- Document automatic isolated-instance cleanup for server-managed
worktrees.
## Verification
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/workspace-instance-cleanup.test.ts` — 9 tests passed.
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/workspace-runtime.test.ts -t "records teardown and cleanup
operations when a recorder is provided"` — 1 test passed and 99 tests
skipped.
- `node scripts/__tests__/provision-worktree-self-heal.test.mjs` — 4
tests passed.
- `bash -n scripts/provision-worktree.sh` — passed.
- `pnpm --filter @paperclipai/server build` — passed.
- `git diff --check` — passed.
## Risks
The main risk is removal of the wrong instance directory. Provisioning
assigns a path-derived ID with a SHA-256 suffix, and cleanup requires
that exact ID in addition to a safe instance identifier, an absolute
configured home, a strict child path, canonical path checks, and a
second canonical path check immediately before removal. It refuses
legacy or mismatched IDs, symlink escapes, and all paths outside the
managed worktree instance root. Cleanup failures become visible warnings
and do not delete an unverified path.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
OpenAI Codex with GPT-5. The runtime does not expose the exact model
snapshot or context-window size. The agent used reasoning, repository
tools, GitHub tools, and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Paperclip <paperclip@paperclip.ing>
Dotta