Commit Graph

4 Commits

Author SHA1 Message Date
Dotta b1efd65edc
fix: continue interrupted task conversations with bounded retries (#13237)
## Thinking Path

> - Paperclip manages AI agents and their tasks.
> - A task can outlive a provider process or a server restart.
> - Legacy recovery treated unknown tool outcomes as a permanent
execution hold.
> - That hold could also reject a later user message.
> - A conversation turn can use prior history without replaying prior
tool calls.
> - This pull request lets supported conversation adapters continue
within the existing retry budget.
> - Users can send a new message after automatic attempts stop.

## Linked Issues or Issue Description

**What happened?**

A server restart could interrupt a local ACP run and leave its task
behind a permanent recovery hold. A later user message could be
cancelled before the provider answered. The immediate recovery path
could also create a successor outside the durable failure counter.

**Expected behavior**

Continue with a bounded new conversation turn. Preserve a compatible
provider session or use full task context when it is unavailable. Do not
replay recorded tools. When automatic attempts stop, allow a new user
request through the normal execution gates.

**Steps to reproduce**

1. Start a task with a local conversation adapter.
2. Restart the server while the provider is working.
3. Let the previous run become interrupted.
4. Send a follow-up message and observe the recovery hold on the old
behavior.

Related work: Refs #13075 for durable task recovery. Refs #12946 for
retry-limit and checkout-lock handling. This change routes conversation
recovery through the existing bounded scheduler.

## What Changed

- Mark supported local conversation failures for continuation. Keep
native-runner and non-conversation recovery rules.
- Carry an interruption notice into the next turn. Retain stopped ACP
session history even when a write outcome is unknown.
- Clear unavailable ACP sessions so the next bounded attempt can use
full task context.
- Route immediate failure recovery through the same durable scheduler as
process-loss recovery. Release only the predecessor checkout when its
retry takes ownership.
- Retire obsolete conversation holds using immutable run evidence, in
bounded batches with an activity record. Preserve outcome evidence and
do not wake historical tasks.
- Block actual admission and Resume while a predecessor process or
environment lease is still active. Keep the original interruption notice
after a rejected wake. Preserve the upstream blocked-wake waiting
contract: bounded retry planning can happen during cleanup, while
deferred messages and execution remain gated.
- Add subprocess and database regression tests. Update the execution
contract.
- Add the current thread-status field to the native recovery provider
fixture so its damaged-journal test reaches the intended boundary.
Tolerate an already-exited fixture process during test cleanup while
still asserting both processes terminate.

## Verification

- Workspace typecheck passed: `pnpm -r typecheck`.
- Build passed: `pnpm build`.
- Module boundaries passed: `pnpm check:module-boundaries`.
- Focused tests passed: 293 recovery/session/dispatch tests, 66 retry
and response-gate tests, and 37 native-session tests. Some suites
overlap.
- Tests cover interrupted writes, missing sessions, concurrent retries,
restart persistence, pending questions and approvals, execution gates,
and historical holds.
- Built the Rust test executables with `pnpm --filter
@paperclipai/paperclip-runner build:rust` for native-runner
verification.
- Full Vitest coverage verified locally using the repository’s general
and serialized shards, with focused reruns for failures and files not
reached after a shard stopped. The ownership-gate regression is fixed
and the complete affected server shard passes (1,390 tests). Local
parallel runs also hit temporary-directory, resource, and timing
failures; those suites pass with canonical temporary paths and
sequential reruns. No test timeouts were increased.
- Final merged-branch regression run: 577 tests pass across process
recovery, retry scheduling, liveness, durable chat, wake-queue
application/adapter, dispatch, continuation, native sessions, and task
chat. Earlier focused verification also passed 19 native control tests.
Token gates and whitespace validation pass.
- Browser verification passed all three ACP Stop/continue/pause
scenarios, including a rerun after merging the upstream waiting
behavior: `PAPERCLIP_E2E_PORT=3397 pnpm test:e2e
tests/e2e/acp-stop-continuation.spec.ts`. The interrupted-write case
verifies that follow-up completes without a repeated write.

- Final-head [CI run
34625037394](https://github.com/paperclipai/paperclip/actions/runs/34625037394)
passed on `06ac4bd9d150f8b209a96e5fd609c696958794a0`: all 31 reported
checks are green, including server/workspace suites, all browser shards,
native runner verification, build, typecheck, release dry run, and
aggregate gates. The two conditional Storybook checks were skipped.
Greptile reviewed this exact commit at 5/5; all review threads are
resolved.

## Risks

- A new model turn can choose to repeat an action. Paperclip does not
replay recorded tool calls and does not certify unknown action outcomes.
- Conversation adapters now stop after their retry budget instead of
requiring action reconciliation. Explicit Stop, pause, dependency,
approval, budget, and ownership gates remain in force.
- No schema migration or dependency changes. Historical holds are folded
without changing task status or waking work.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools, code
execution, and test execution. The session does not expose a more
specific model build ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-11 12:16:04 -05:00
Dotta eb640ec129
fix(execution): keep blocked wakes waiting without repeated runs (#13236)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Wake admission decides when a task can create an execution run.
> - Recovery can prohibit replay while the previous execution needs
review.
> - Dependency reconciliation kept creating runs before dispatch
rejected that same hold.
> - Each rejected run added another startup notice without doing useful
work.
> - This change checks the hold during admission and records repeated
automatic waits once.
> - Tasks keep their messages and can resume when the current gates
permit execution.

## Linked Issues or Issue Description

Related changes: Refs #13173 (stale completed-task continuations). Refs
#12651 (dependency waits during recovery).

**What happened?**

A blocked task with completed dependencies can remain under a durable
execution reconciliation hold. Each scheduler pass created a queued run.
Dispatch then cancelled it before the adapter started. The skipped wake
did not satisfy dependency wake deduplication, so this repeated and
filled the conversation with “Couldn't start” notices.

**Expected behavior**

A known execution hold creates a waiting diagnostic without a run.
Repeated automatic observations share that diagnostic. Clearing the hold
permits a new wake only after the other gates pass. New comments remain
available for the next eligible execution.

**Steps to reproduce**

1. Assign a blocked task with a completed blocker.
2. Give the task an active reconciliation action, or a resolved action
whose automatic recovery evidence still prohibits replay.
3. Run dependency reconciliation repeatedly.
4. Observe repeated cancelled pre-start runs on the base branch. This
branch creates no runs while held and admits work after the effective
hold clears.

## What Changed

- Check effective execution holds under the issue admission lock before
inserting runs. Keep the final dispatch check for races.
- Share automatic wait diagnostics across producers, wake keys, and
service restarts. Apply the helper to reconciliation, dependencies,
pause holds, availability, budgets, and disabled heartbeats.
- Preserve ordinary comment and interaction receipts during execution
holds. Prevent release from draining them while replay is blocked. Keep
external-chat receipt authorization intact.
- Group empty pre-start reconciliation cancellations into a neutral
waiting notice. Keep started runs and the full run history.
- Document the waiting contract and add database-backed, UI, and browser
regressions.
- Stabilize two existing verification tests: allow the asynchronous chat
lease transition a bounded five-second wait, and accept either
legitimate damaged-session refusal while retaining exact
archive-evidence assertions.

## Verification

Passed targeted tests:

- `pnpm exec vitest run
server/src/__tests__/heartbeat-issue-liveness-escalation.test.ts
server/src/modules/wake-queue/adapters/postgres.test.ts` — 28 tests.
- `pnpm exec vitest run ui/src/components/TaskChatThread.test.tsx` —
covered in the initial combined test run; UI suite passed.
- Run-dispatch adapter tests passed in the combined gate regression run.
- `pnpm exec vitest run
server/src/__tests__/durable-chat-wakeup.test.ts` — 41 tests, including
held receipt replay, promotion, and revoked access.
- `PAPERCLIP_E2E_PORT=3294 pnpm test:e2e
tests/e2e/acp-stop-continuation.spec.ts` — all 3 browser scenarios pass.
Repeated held messages create no additional runs or provider prompts and
do not replay writes.
- `pnpm check:token-gates`
- `pnpm check:module-boundaries`
- `git diff --check`

`pnpm -r typecheck` and `pnpm build` pass.

The full local `pnpm test:run` invocation did not finish green: it
encountered exhausted local PostgreSQL shared-memory slots, a missing
fresh-worktree runner test binary, and tests loaded across in-flight
edits. The affected chat/database suites passed on rerun (81 tests), and
the targeted lifecycle/recovery verification passed (3 tests). After
building the runner test binary, the full native session suite also
passed (37 tests). Final-head [CI run
34621288475](https://github.com/paperclipai/paperclip/actions/runs/34621288475)
passed on `8659618b0ed2b98df002a28f4c1bd97321b0db04`, including all
server/workspace test shards, all three browser shards, runner
verification, typecheck, build, release dry run, and the aggregate
verification gates. All 31 reported checks passed; the two conditional
Storybook checks were skipped as intended. Greptile reviewed that exact
commit at 5/5 with no unresolved review threads.

## Risks

The wait record is diagnostic only. It must never count as a delivered
wake or bypass a current gate. Tests cover repeated and concurrent
admission, resolved no-replay evidence, a remaining dependency after
hold clearance, deferred comments, and release gating. Explicit user
requests and authorized chat receipts do not share automatic
diagnostics. No migration or historical data deletion is required.
Existing provider retry budgets remain unchanged.

## Model Used

OpenAI GPT-6 through Codex. The runtime does not expose the exact hosted
snapshot ID or context-window size. Used repository inspection, code
editing, command execution, tests, and review tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-11 11:43:40 -05:00
Dotta 52811c6ce6
fix(tasks): require resume before sending to paused tasks (#13232)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Task execution controls let board users pause a task or its subtree.
> - The composer still accepted messages while a pause hold was active.
> - A paused task must require an explicit resume before the user can
send another message.
> - This pull request replaces the composer with an amber pause card and
checks board comment writes on the server.
> - The user keeps their draft and resumes through the existing task
controls.

## Linked Issues or Issue Description

Refs #13104. Refs #13119.

**What existing behavior does this improve?**

The task composer and existing task/subtree pause controls.

**Current behavior**

A paused task can still receive a board message. The pause notice sits
outside the composer, which leaves the send action available.

**Proposed behavior**

Show an amber takeover in both task chat and the classic composer.
Preserve the draft. Require the user to resume the task or the ancestor
subtree before sending. Reject board comment writes through either
supported write route while the pause hold is active.

**Breaking changes**

Board comment writes to a paused task now return HTTP 409. Agent run
reports remain supported during a pause. There is no schema migration.

## What Changed

- Add a shared amber composer takeover with task, subtree, saved draft,
pending, and error states.
- Use effective ancestor pause state in both composer interfaces.
Refresh it after pause events, task updates, and rejected sends.
- Preserve draft text and attachments. Hide editor, send, queued edit,
and pending question controls while paused.
- Check active pause holds before board comment writes can mutate tasks,
store comments, or wake agents.
- Connect the approved Storybook examples to the production component
and update the design and behavior docs.
- Add browser coverage for both composers, draft persistence, resume,
inherited holds, and rejected writes. Update ACP continuation coverage
for the explicit resume requirement.

## Verification

- Passed: `pnpm -r typecheck`.
- Passed: `pnpm build`.
- Passed: `pnpm build-storybook`.
- Passed: `pnpm check:token-gates` and `git diff --check`.
- Passed: focused UI tests (398 tests) and server route tests (127
tests).
- Passed: `pnpm exec playwright test --config
tests/e2e/playwright.config.ts tests/e2e/paused-composer.spec.ts
tests/e2e/acp-stop-continuation.spec.ts` (5 tests).
- Passed: manual browser walkthrough in a disposable local instance.
Pause with a draft, refresh while paused, resume, send, and reopen. The
draft returned, and one message persisted. The amber card and resume
dialog were readable with no clipping.
- Full local `pnpm test:run` did not pass: the general-server stage
recorded 9,072 passing tests, 6 database setup failures from macOS
shared-memory exhaustion, and 4 failed tests. This stopped the script
before its later groups. Latest-head CI runs those groups independently.
- Local follow-up: the Git file-resource load test passed on rerun (4
tests); native finalization migration passed after clearing the
abandoned browser-test database allocation. Building the native debug
fixtures fixed the missing fake provider. The remaining native-session
recovery assertion also reproduces on untouched base commit `87b3e5fc6`
(36 pass, 1 fail on both base and PR). It expects a settled-session
error but receives a semantic-input-digest error.
- The final UI build, UI typecheck, token gates, both thread suites (182
tests), and all five browser tests passed after the queued-action review
fix. All 31 latest-head CI checks passed, including all server,
workspace, browser, build, release, and security gates. Two optional
Storybook jobs were skipped by workflow policy. Greptile reviewed
`32d8fb5f5` at 5/5 with no open findings.
- Review the Paused Composer and Tasks / Execution Controls stories.
Pause a task with a draft, verify the amber card, resume, and verify the
draft can be sent once.

## Risks

- Clients that used board comments to continue paused work must resume
first. The response is an explicit HTTP 409.
- Pause state can change while a page is open. Live updates refresh the
composer, and the server rejects stale sends before their side effects.
- Resume keeps the existing dialog and optional agent wake behavior.
Agent reports from interrupted runs remain allowed.

## Model Used

OpenAI Codex, based on GPT-6, assisted with design, implementation, code
execution, and browser verification. The exact runtime model ID and
context window are not exposed in this session. The agent used reasoning
and tool calls.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run the relevant tests locally and they pass; the full
local-suite limits are documented above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-11 10:28:00 -05:00
Dotta 018ca5daaf
fix: verify ACP Stop and preserve safe continuation (#13119)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Task controls coordinate provider execution and queued user
messages.
> - Stop could finish before an embedded ACP provider stopped its tools.
> - A later request could be held for reconciliation without a clear
task response.
> - A restored provider could also retain the stopped run's API
credential.
> - This pull request verifies provider termination and preserves safe
session continuation.
> - Operators can continue known-safe work and see why uncertain work
cannot start.

## Linked Issues or Issue Description

**What happened?**

Stop could leave an embedded ACP provider running. A queued follow-up
followed by “go” could fail before it reached the provider. Task chat
could show a generic missing-response message. Even a restored session
could use the previous run's credential and fail its task update.

**Expected behavior**

Stop waits for confirmed provider termination. A later explicit wake
continues the same compatible session only when recorded actions have
known outcomes. It carries pending comments and the current run's
environment. Uncertain actions retain a visible reconciliation hold.
Composer Stop preserves the existing pause rule: conversation can
continue while paused, but task work requires Resume.

**Steps to reproduce**

1. Start an embedded ACP task.
2. Send a second request while the provider is running.
3. Interrupt the run, then send “go”. Also test composer Stop followed
by Resume work.
4. Check that the request is delivered once and that the provider can
complete the task through the current run's API credential.
5. Repeat with an unfinished write. Confirm that the write stops and
that further execution stays blocked with a visible reason.

**Paperclip version or commit**

Built from source on master at `3bc60dd8b` plus this branch.

**Deployment mode**

Local source build with an isolated embedded PostgreSQL instance.

Refs #11183. Refs #12552. Those changes address recovery after operator
cancellation. This change also covers embedded ACP termination, session
proof, pending-comment delivery, and task feedback.

## What Changed

- Propagate Stop into embedded ACP and wait for bounded adapter cleanup
and provider exit. Retain the actual ChildProcess object for forced
termination on all platforms; never signal a recycled numeric PID.
- Preserve interrupted checkpoints only for acknowledged, local,
persistent sessions with settled reads or no tools. Keep writes,
incomplete actions, and forced termination blocked.
- Restore the same compatible provider session with the current run's
environment. Reject fresh-session fallback for an interrupted
checkpoint.
- Adopt pending comments on the next explicit wake. Stop alone does not
dispatch them.
- Share the execution-blocker rule across dispatch, Resume, and task
detail. Show Stopped or Couldn't start with the recorded reason. Resolve
the stopped agent for the run link, including reviewer runs.
- Keep execution reconciliation holds intact when generic recovery sees
queued comments or healthy child tasks.
- Add process, service, component, and browser regression coverage. Fix
disposable database cleanup and React test settling exposed by the full
suite.

## Verification

- Passed `pnpm -r typecheck`, `pnpm build`, and `pnpm
check:token-gates`.
- Passed all three `acp-stop-continuation.spec.ts` browser journeys.
They use an actual ACP child process and require task completion through
the agent API.
- Passed 165 adapter execution, operator-stop, and child-process control
tests, 17 queued-comment route tests, and 65 tests in the two adjusted
UI suites. Earlier focused recovery, heartbeat, and task-control tests
also passed.
- Manually used the browser to queue a request, Stop, send “go” while
paused, and Resume. The same session answered once and moved the task to
Done with the current run's credential.
- Manually interrupted an unfinished write. Its file size stayed fixed
for five seconds. “Go” showed the reconciliation reason and did not
start another provider prompt.
- Separate live Claude ACP smoke checks confirmed that Stop ended a
disposable local write and that a no-tool interruption could resume the
exact provider session. The browser fixture does not call Drive or
another external app.
- Passed all 5,615 UI tests and 3,090 other workspace tests. The CLI and
general server groups pass with targeted retries: two transient server
failures passed together on retry, and two embedded-database startup
failures passed after removing abandoned shared-memory segments from
this task's completed browser fixtures. All 144 serialized server suites
completed, with 2,189 tests passing after two transient HTTP socket
failures passed on retry.
- Passed all 135 heartbeat process/recovery tests, including a
deterministic regression that failed before the recovery-sweep fix.
- Passed 18 dispatch integration tests, including stopped-reviewer
links, company boundaries, and malformed run IDs.
- Greptile is 5/5 on `7dd170d83`, with zero unresolved review threads.
The security scan and all required CI gates pass for the same commit.

## Risks

- Safe continuation depends on complete tool reporting and a restorable
local provider session. Unknown outcomes remain blocked and require
reconciliation.
- Provider cleanup can take time. A timeout does not grant replay
permission.
- The change adds optional adapter context fields and an optional issue
projection. It does not change the database schema or require a
migration.
- Test cleanup truncates company data only in a disposable test
database.

## Model Used

OpenAI GPT-6, running as Codex with repository tools, code execution,
and browser interaction. The runtime does not expose a more specific
model deployment ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-09 22:06:06 -05:00