## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents run inside environments. A sandbox environment gets its
sandbox from a provider plugin (for example the bundled
`paperclip.kubernetes-sandbox-provider`), and every run starts by
acquiring a lease through that plugin.
> - When a plugin activation fails once (on a hosted deployment: one
`RPC call "initialize" timed out after 15000ms`), the loader calls
`markError`. That persists `status = error` on the plugin row and
switches off worker auto-restart. Boot activation (`loadAll`), the
bundled-plugin bootstrap and the lazy worker recovery all consider only
`ready` plugins, so the plugin stays in `error` across restarts until an
operator enables it by hand.
> - Every run that needs the provider then fails before dispatch with
`Sandbox provider "kubernetes" is installed via plugin "...", but that
plugin is currently error.` That message matches neither the retryable
classifier (`... but its worker is not running`) nor any configuration
classifier, so the run is recorded as a plain `setup_failed`, the issue
is released, and the scheduler dispatches the same failing run again on
the next tick. On the hosted deployment one company produced about
11,300 identical failed runs, one every 30 seconds, for a week (#12953
is a customer's report of the same condition).
> - Two gaps cause this: the heartbeat treats a condition that only an
operator can change as a transient setup failure, and the bundled-plugin
bootstrap never gives a plugin in `error` another chance even though the
bundle ships with the release image.
> - This pull request classifies the "installed but not ready" lease
failure as `configuration_incomplete`, so the existing recovery path
moves the issue to `blocked` with one recovery action and an actionable
notice; and it re-enables a bundled plugin found in `error` once per
boot, so the next server restart heals the plugin.
> - The benefit is that a stuck provider plugin surfaces as one blocked
issue per task with clear next steps, instead of an endless stream of
identical failed runs, and a restart repairs the plugin without an
operator having to know the plugin API.
## Linked Issues or Issue Description
- Refs #12953 — hosted report: "that plugin is currently error" on every
run for six days, including runs that were retried by hand. This PR
stops the retry loop (issue goes to `blocked`) and makes a server
restart re-activate the bundled plugin. It does not change how a managed
Kubernetes environment is provisioned for a company, which the same
report also mentions.
- Related PR: #9760 pauses the agent for the permanent `Adapter "..." is
not in the configured adapter registry` setup failure. This PR handles a
different permanent condition (plugin not `ready`) and routes it through
the existing `configuration_incomplete` recovery path (issue-level block
with a recovery action) rather than an agent-level pause, because the
gap is on the plugin, not on the agent. The two do not overlap in code
paths.
- No existing issue covers the bundled-plugin re-enable. Bug
description:
**What happened**
A bundled sandbox provider plugin went to `status = error` after one
failed activation. It stayed in `error` across every later server
restart. Every run for every agent on that provider failed lease
acquisition in under a second with `... but that plugin is currently
error.` (`setup_failed`), and the heartbeat kept dispatching new runs
that failed the same way.
**Expected behavior**
A run that fails because its provider plugin is not `ready` is recorded
as a configuration gap and the issue is moved to `blocked` with a notice
that names the plugin and its status, so no further runs are dispatched
until an operator acts. A bundled plugin left in `error` gets a fresh
activation attempt on the next boot.
**Steps to reproduce**
1. Install a sandbox provider plugin and create a sandbox environment
that uses it; make it an agent's default environment.
2. Set the plugin row's status to `error` (or make its worker fail
`initialize` once so the loader does it).
3. Assign an issue to the agent and let the heartbeat run it.
4. Observe: the run fails with `... but that plugin is currently error.`
as `setup_failed`, the issue is released, and the next tick dispatches
another run that fails the same way. Restart the server: the plugin is
still `error`.
**Paperclip version**
master at 856813ba3 (`fix(connections): distinguish local setup from
provider handoff (#12947)`).
**Deployment mode**
Hosted (Kubernetes, bundled kubernetes sandbox provider plugin). The
heartbeat behavior is the same in self-hosted mode.
## What Changed
- `server/src/services/heartbeat.ts`
- New exported `parseSandboxProviderPluginNotReadyFailureMessage()`
recognises environment-runtime's `not_ready` lease message (`... is
installed via plugin "<key>", but that plugin is currently
error|disabled|upgrade_pending`) and returns the provider, plugin key
and status. It does not match the transient `... but its worker is not
running` message (still retried) or the permanent "not installed"
message (unchanged).
- In the setup-failure catch, a matched message sets `errorCode =
configuration_incomplete` and records (independently of whether the
agent lookup succeeded) a `configurationIncomplete` payload with
`reason: "sandbox_provider_plugin_not_ready"`, the provider,
`pluginKey`, `pluginStatus`, and a `fingerprint` of
`sandbox_provider_plugin:<key>:<status>`, so repeated failures on the
same stuck plugin reuse one recovery action. The existing recovery flow
then blocks the issue, skips the infra retry, and posts one notice.
- The two places that build the configuration-incomplete notice now pass
the run's payload so the notice can name the specific gap.
- `server/src/services/recovery/stranded-notice.ts`:
`buildConfigurationIncompleteRecoveryNoticeSeed` takes the optional
payload. For `sandbox_provider_plugin_not_ready` the body names the
plugin and its status and gives status-specific guidance
(`sandboxProviderPluginRemedy`): review and approve the upgraded
capabilities before enabling for `upgrade_pending`, enable again for an
operator `disabled`, enable or restart for `error`. Other reasons keep
the secret/env-binding wording. Exports
`SANDBOX_PROVIDER_PLUGIN_NOT_READY_REASON`.
- `server/src/services/recovery/service.ts`: the recovery action's
`nextAction` for this reason uses the same status-specific guidance
instead of "bind the missing secret(s)". Small refactor:
`readConfigurationIncompletePayload` backs the existing fingerprint
reader.
- `server/src/services/bundled-plugins.ts`
- `ensureBundledPlugins` no longer skips a present bundled plugin whose
status is `error`. It logs at `warn` with the row's `lastError`, resets
the row to `ready` with `lastError` cleared through
`registry.updateStatus` (a plain status reset, not `lifecycle.enable()`,
so no `plugin.enabled` event fires before the worker runs; the startup
`loadAll()` that follows does the activation and its events), and
continues boot on failure. This runs once per boot by construction; if
activation fails again the loader marks `error` again and nothing
retries until the next boot.
- `installed`, `ready`, `disabled` and `upgrade_pending` rows are still
skipped, so an operator's `disabled` stays untouched.
- `BundledPluginProvisionerDeps` gains `registry.updateStatus` and
`logger.warn`; `app.ts` already passes objects that have both.
- `doc/plugins/PLUGIN_SPEC.md`: one bullet in 12.4 Failure Policy about
the once-per-boot re-enable of bundled plugins.
- Tests
- `server/src/__tests__/bundled-plugins.test.ts`: re-enables an `error`
row exactly once with the `lastError` in the warn log and no reinstall;
continues boot and provisions later entries when `enable` throws; still
skips `installed`/`ready`/`disabled`/`upgrade_pending` without calling
`enable`.
- `server/src/__tests__/heartbeat-process-recovery.test.ts` (embedded
PostgreSQL): a plugin row in `error` plus a sandbox environment produce
a run with `errorCode = configuration_incomplete` and the expected
payload, the adapter is never dispatched, no retry or second run is
created, the issue is `blocked`, the recovery action is
`configuration_validation` with the plugin next action, and the notice
names the plugin key and status. Plus a unit case for the message parser
(positive for the three statuses and a wrapped message, negative for
both other sandbox messages).
- `server/src/services/recovery/stranded-notice.test.ts`: the
plugin-specific body, and the unchanged secret-binding body for other
reasons.
## Verification
- `cd server && pnpm typecheck` — passes.
- `cd server && pnpm exec vitest run
src/__tests__/bundled-plugins.test.ts` — 29 tests pass.
- `cd server && pnpm exec vitest run src/services/recovery/` — 77 tests
pass (includes the stranded-notice and classification suites).
- `cd server && pnpm exec vitest run
src/__tests__/heartbeat-process-recovery.test.ts -t "sandbox
provider|retryable pattern|secret ref has no binding"` — 5 tests pass:
the two new cases, the existing transient worker-restart retry, the
existing non-retryable "not installed" escalation, and the existing
secret-binding `configuration_incomplete` block (embedded PostgreSQL).
- Manual check for a reviewer: set a sandbox provider plugin row to
`status = 'error'`, run an agent on that provider, and confirm the issue
moves to `blocked` with a "Configuration incomplete" notice that names
the plugin, and that no second run appears. Restart the server and
confirm the boot log shows `bundled plugin is in error status from a
previous activation; re-enabling it for this boot` followed by normal
activation.
## Risks
- Behavior change: a run against a plugin in `error`, `disabled` or
`upgrade_pending` now blocks the issue instead of failing as
`setup_failed` and being re-picked. For `disabled` this is deliberate:
an operator switched the plugin off, and re-dispatching cannot help. The
block is reversible from the issue (retry or reassign) like every other
`configuration_incomplete` block.
- The classifier is anchored on the exact `... but that plugin is
currently <status>` phrase from `environment-runtime.ts`. If that
message changes, the run falls back to the previous `setup_failed`
behavior (no worse than today). A unit test pins the phrase.
- Bundled re-enable: a bundled plugin whose activation fails on every
boot now costs one activation attempt (the `initialize` timeout, 15 s by
default) per boot instead of none. It runs inside the existing
non-awaited bootstrap chain, so boot time is unaffected. Non-bundled
plugins are untouched.
- No migration, no schema change. The `configurationIncomplete` payload
is JSON in `heartbeat_runs.result_json`, read only by the recovery
service.
## Model Used
- Claude Fable 5.1 (`claude-fable-5-1`) via Claude Code, extended
thinking, tool use (file edits, shell, test runs). The change was
produced with the model and reviewed by the submitting human.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_014t3bi2beVNVVHAxK36dmXm
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
5752d6bd93