fix: recover sandbox workspace setup and retries (#13353)
## Thinking Path > - Paperclip manages AI agents and their tasks. > - Sandbox tasks need a workspace and a provider session before work can start. > - A selected Git subfolder is a valid workspace, but it is not a repository fetch source. > - A resumed sandbox must keep one resource identity when the provider fills in its region. > - Workspace reuse does not prove that a provider session has started. > - A recorded continuation must not leave an old failure blocking Retry for a newer attempt. > - This pull request fixes those setup and recovery boundaries while preserving ownership checks. ## Linked Issues or Issue Description **What happened?** A Daytona task failed before provider startup when its project folder was inside a parent Git repository. After the folder was repaired, Retry resumed the same sandbox and verified its workspace sentinel, but workspace preparation reported that the lease was no longer active. A later attempt could fail because the resumed workspace had no provider checkpoint. The old run also kept a recovery projection after the server recorded its explicit successor, hiding Retry for the new failure. **Expected behavior** Sync the selected folder without importing parent files or history. Keep the existing sandbox identity stable. Allow a new provider session only with proof that its exact session has never started. Let the current failed attempt retain Retry once the old recovery has a recorded successor. **Steps to reproduce** 1. Select a subfolder of a Git repository as a project workspace and start a Daytona task. 2. Leave the target region unset. Release its reusable sandbox after setup fails, then resume and realize the workspace. 3. Retry a provider setup that failed before any session directory or checkpoint was created. 4. Record an explicit successor for a native failure, fail that successor during setup, and inspect Retry in the task thread. **Paperclip version or commit** Reproduced against `8d1f0c20a` with new failing regressions before each production fix. **Deployment mode** Local controller with a Daytona environment. Automated tests use deterministic provider fixtures and real filesystem operations. Related: #13338, #13163, #13264, #13349. The open work-folder stack in #13264 includes a broader fresh-session authority change. This patch addresses the independently reproduced startup failure with existing durable bootstrap proof and atomic directory creation. It does not include the work-folder migration or credential changes from that stack. ## What Changed - Classify only the selected Git repository root as a fetch source. Sync subfolders as directories and preserve their enclosing Git ignore rules on upload and restore. Recognize the shared scheduler’s completed non-repository result so ordinary folders still sync; timeout, cancellation, and output-limit failures remain closed. - Remove the placement target from Daytona account cache identity. Keep API endpoint, credential digest, company, environment, driver, and sandbox ID boundaries. - Preserve closed-lease admission until a sentinel-verified resume reopens that same resource. - Show the already-recorded explicit successor of a resolved recovery. Keep the old failure evidence and unresolved holds. No historical status writes occur. - Permit a resumed workspace to create a new session directory only with matching durable identity, zero connections and events, untouched bootstrap commands, no backup, and an absent remote session. Claim the directory atomically. Existing, partial, or ambiguous state still blocks startup. ## Verification - Final head `c43c7403f`: [CI completed successfully](https://github.com/paperclipai/paperclip/actions/runs/34733846820/attempts/2), with 32 successful checks and 2 conditional skips. This includes every server, UI, package, serialized-route, browser, native-runner, typecheck, and build gate. Greptile reviewed the same head at [5/5 with no unresolved findings](https://github.com/paperclipai/paperclip/pull/13353#issuecomment-5650270168). - New regressions failed before each of the four production fixes. Git/archive/restore suites: 148 passed. The scheduler-wrapped non-Git regression also failed before its fix; 135 affected Git/sync/Codex tests then passed. - Daytona plugin: 230 passed, 6 opt-in live tests skipped. Native executor, projection, and TaskChatThread: 494 passed, including existing/partial state, wrong identity, prior connections or turns, backups, unavailable proof, and unresolved recovery controls. - Local full-repository typecheck, build, and token gates passed on the final head. Local CLI: 485 passed. Complete single-worker package rerun: 3,224 passed, 19 skipped. - Local verification is an aggregate with recorded retries, not one pristine green invocation: the general-server run began on `e721a920a` and finished with 12,002 passed, 3 failed, 70 skipped. Its real Codex scheduler failure is fixed above; the socket and workspace-runtime timeout failures passed unchanged in focused reruns. Both Inbox failures passed unchanged in the full 27-test Inbox file; database/shared-package failures passed in the single-worker package rerun. The supplemental local serialized-route rerun remains in progress; all five corresponding final-head CI lanes passed. - The first final-head CI attempt hit a Daytona fixture-readiness race and a signoff-browser heartbeat receipt timeout. One supported unchanged failed-job rerun passed both and the aggregate gates. Live combined user-journey verification is tracked in the related follow-up; this PR's provider tests use deterministic fixtures and real filesystem checks. ## Risks - A selected subfolder uses directory sync and does not carry parent Git history. Its ignored files stay local. - The target region remains a creation setting and part of workspace reuse policy; it does not split the account identity of an existing sandbox. - Incomplete or conflicting provider state still fails closed. This change does not erase a session, infer completed work, bypass a user decision, or replay uncertain actions. - A later remote setup failure can leave a claimed partial session directory. It remains blocked rather than being overwritten. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and code execution. The exact hosted model ID and context-window size are not exposed in this task. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Dotta
GitHub
parent
8d1f0c20af
commit
6809314a3f
|
|
@ -606,6 +606,8 @@ Paperclip applies one process-wide scheduler to expensive host-side workspace Gi
|
|||
|
||||
The cache intentionally trades up to a few seconds of changed-file freshness for stable server latency. The file browser retains an explicit refresh action, does not start its query while the panel or browser tab is hidden, and presents overloads as retryable failures rather than an empty workspace. A full queue returns `503` with code `workspace_git_scan_saturated`; a scan exceeding its wall-clock limit returns `504` with code `workspace_git_scan_timeout`. Both responses include `Retry-After: 1`.
|
||||
|
||||
Sandbox Git sync treats only the selected repository root as a clone source. A selected subfolder uses directory sync within that folder, applies the enclosing repository's ignore rules, and does not transfer parent files or Git history.
|
||||
|
||||
Environment overrides:
|
||||
|
||||
- `PAPERCLIP_WORKSPACE_GIT_SCAN_CONCURRENCY` (default `2`, range `1`–`16`)
|
||||
|
|
@ -990,6 +992,8 @@ that classification finishes.
|
|||
Codex thread and goal state. Empty retry directories do not prevent recovery;
|
||||
conflicting histories, changed profiles, and live or unverifiable owners do.
|
||||
|
||||
A resumed sandbox lease can contain a workspace whose provider never started. A new attempt may create its exact session directory only when durable control-plane evidence proves zero connections, zero events, and untouched bootstrap commands, and no backup or remote session directory exists. Directory creation is atomic; partial state or uncertain ownership remains blocked.
|
||||
|
||||
Run the credential-free real-process restart suite with:
|
||||
|
||||
```sh
|
||||
|
|
|
|||
|
|
@ -118,6 +118,19 @@ describe("git workspace sync", () => {
|
|||
return repo;
|
||||
}
|
||||
|
||||
it("does not classify a selected repository subfolder as a cloneable repository root", async () => {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-selected-folder-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
const repo = await createRepo(rootDir);
|
||||
const selectedDir = path.join(repo, "project");
|
||||
await mkdir(selectedDir);
|
||||
await writeFile(path.join(selectedDir, "draft.md"), "selected work\n");
|
||||
|
||||
expect(await git(selectedDir, ["rev-parse", "--is-inside-work-tree"])).toBe("true");
|
||||
expect(await readGitWorkspaceSnapshot(selectedDir)).toBeNull();
|
||||
expect((await readGitWorkspaceSnapshot(repo))?.headCommit).toBe(await git(repo, ["rev-parse", "HEAD"]));
|
||||
});
|
||||
|
||||
it("creates a shallow standalone clone from the local HEAD snapshot", async () => {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-sync-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
|
|
@ -588,6 +601,27 @@ describe("git workspace sync", () => {
|
|||
await expect(readReferencedSourceGitIgnoredPaths(plainDir)).resolves.toBeNull();
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ code: "workspace_git_scan_failed", exitCode: 128, signal: null, nonGit: true },
|
||||
{ code: "workspace_git_scan_timeout", exitCode: 128, signal: null, nonGit: false },
|
||||
{ code: "workspace_git_scan_cancelled", exitCode: 128, signal: null, nonGit: false },
|
||||
{ code: "workspace_git_scan_output_limit", exitCode: 128, signal: null, nonGit: false },
|
||||
{ code: "workspace_git_scan_failed", exitCode: null, signal: "SIGTERM", nonGit: false },
|
||||
])("classifies scheduled non-repository failures without swallowing $code/$signal", async ({ code, exitCode, signal, nonGit }) => {
|
||||
const error = Object.assign(new Error("Workspace Git scan failed"), {
|
||||
code,
|
||||
details: { exitCode, signal, stderr: "fatal: not a git repository (or any of the parent directories): .git" },
|
||||
});
|
||||
setExpensiveWorkspaceGitExecutor(async () => { throw error; });
|
||||
try {
|
||||
const result = readReferencedSourceGitIgnoredPaths("/plain-workspace");
|
||||
if (nonGit) await expect(result).resolves.toBeNull();
|
||||
else await expect(result).rejects.toBe(error);
|
||||
} finally {
|
||||
setExpensiveWorkspaceGitExecutor(null);
|
||||
}
|
||||
});
|
||||
|
||||
it("reads the repository top level and the ignored paths of a Git work tree", async () => {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-referenced-git-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
|
|
|
|||
|
|
@ -146,6 +146,19 @@ export async function readGitWorkspaceSnapshot(localDir: string): Promise<GitWor
|
|||
return null;
|
||||
}
|
||||
|
||||
const toplevelResult = await runLocalGit(localDir, ["rev-parse", "--show-toplevel"], {
|
||||
timeout: 10_000,
|
||||
maxBuffer: 16 * 1024,
|
||||
});
|
||||
// Git discovers a parent repository from a nested project directory, but
|
||||
// that directory is not a fetch source. Keep the selected workspace
|
||||
// boundary: subfolders use directory sync instead of importing the parent.
|
||||
const [workspacePath, repositoryPath] = await Promise.all([
|
||||
fs.realpath(localDir),
|
||||
fs.realpath(toplevelResult.stdout.trim()),
|
||||
]);
|
||||
if (workspacePath !== repositoryPath) return null;
|
||||
|
||||
const [headCommitResult, branchResult, overlayDiffResult, untrackedResult, deletedResult, ignoredResult] = await Promise.all([
|
||||
runLocalGit(localDir, ["rev-parse", "HEAD"], {
|
||||
timeout: 10_000,
|
||||
|
|
@ -278,6 +291,15 @@ async function runHardenedReadOnlyGit(
|
|||
* surface as a failure and never look like "no Git tree here".
|
||||
*/
|
||||
function isNotAGitRepositoryError(error: unknown): boolean {
|
||||
// The host scheduler keeps bounded subprocess diagnostics under details.
|
||||
// Only a completed Git exit may establish that no repository exists.
|
||||
if (error && typeof error === "object" && "code" in error &&
|
||||
typeof error.code === "string" && error.code.startsWith("workspace_git_scan_")) {
|
||||
const details = "details" in error && error.details && typeof error.details === "object"
|
||||
? error.details as Record<string, unknown> : {};
|
||||
return error.code === "workspace_git_scan_failed" && details.exitCode === 128 && details.signal === null &&
|
||||
typeof details.stderr === "string" && /not a git repository/i.test(details.stderr);
|
||||
}
|
||||
const stderr = error && typeof error === "object" && "stderr" in error ? String((error as { stderr: unknown }).stderr) : "";
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
return /not a git repository/i.test(stderr) || /not a git repository/i.test(message);
|
||||
|
|
|
|||
|
|
@ -92,6 +92,40 @@ describe("sandbox native file sync", () => {
|
|||
}
|
||||
});
|
||||
|
||||
it("syncs a selected repository subfolder without parent files, history, or ignored files", async () => {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-native-nested-workspace-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
const repo = path.join(rootDir, "repo");
|
||||
const selectedDir = path.join(repo, "project");
|
||||
const remoteDir = path.join(rootDir, "remote");
|
||||
await mkdir(selectedDir, { recursive: true });
|
||||
await execFile("git", ["-C", repo, "init"]);
|
||||
await writeFile(path.join(repo, "outside.txt"), "outside boundary\n");
|
||||
await writeFile(path.join(repo, ".gitignore"), "project/private.txt\n");
|
||||
await writeFile(path.join(selectedDir, "draft.md"), "preserved draft\n");
|
||||
await execFile("git", ["-C", repo, "add", "."]);
|
||||
await execFile("git", ["-C", repo, "-c", "user.name=Test", "-c", "user.email=test@example.com", "commit", "-m", "base"]);
|
||||
await writeFile(path.join(selectedDir, "private.txt"), "stay local\n");
|
||||
|
||||
const { client } = makeNativeClient();
|
||||
const prepared = await prepareSandboxManagedRuntime({
|
||||
spec: { transport: "sandbox", provider: "test", sandboxId: "s1", remoteCwd: remoteDir, timeoutMs: 30_000, apiKey: null },
|
||||
adapterKey: "test-adapter",
|
||||
client,
|
||||
workspaceLocalDir: selectedDir,
|
||||
});
|
||||
|
||||
expect(await readFile(path.join(remoteDir, "draft.md"), "utf8")).toBe("preserved draft\n");
|
||||
for (const absent of [".git", "outside.txt", "project", "private.txt"]) {
|
||||
await expect(lstat(path.join(remoteDir, absent))).rejects.toMatchObject({ code: "ENOENT" });
|
||||
}
|
||||
await writeFile(path.join(remoteDir, "draft.md"), "continued draft\n");
|
||||
await prepared.restoreWorkspace();
|
||||
expect(await readFile(path.join(selectedDir, "draft.md"), "utf8")).toBe("continued draft\n");
|
||||
expect(await readFile(path.join(selectedDir, "private.txt"), "utf8")).toBe("stay local\n");
|
||||
expect(await readFile(path.join(repo, "outside.txt"), "utf8")).toBe("outside boundary\n");
|
||||
});
|
||||
|
||||
it("prefers the native path for default-provision asset inbound and workspace outbound", async () => {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-native-sync-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
|
|
|
|||
|
|
@ -1127,7 +1127,17 @@ export async function prepareSandboxManagedRuntime(input: {
|
|||
readGitWorkspaceSnapshot(input.workspaceLocalDir),
|
||||
)
|
||||
: null;
|
||||
const gitIgnoredExcludes = gitSnapshot?.ignoredPaths;
|
||||
// A selected subfolder has no cloneable Git snapshot, but its parent
|
||||
// repository's ignore rules still govern which files may leave the host.
|
||||
// Use the same bounded, path-relative resolver as referenced project trees.
|
||||
const directoryIgnore = syncWorkspace && !gitSnapshot
|
||||
? await resolveReferencedSourceIgnore(input.workspaceLocalDir)
|
||||
: null;
|
||||
if (directoryIgnore?.kind === "failed") {
|
||||
throw new Error(`Workspace ignore scan failed: ${directoryIgnore.reason}`);
|
||||
}
|
||||
const gitIgnoredExcludes = gitSnapshot?.ignoredPaths
|
||||
?? (directoryIgnore?.kind === "git" ? directoryIgnore.ignoredPaths : undefined);
|
||||
const workspaceArchiveExclude = mergeExcludes(
|
||||
SANDBOX_WORKSPACE_HEAVY_DIR_EXCLUDES,
|
||||
[...GIT_ARCHIVE_EXCLUDES],
|
||||
|
|
|
|||
|
|
@ -27,7 +27,7 @@ Notes:
|
|||
|
||||
- The current published Daytona SDK package is `@daytonaio/sdk`.
|
||||
- The driver supports both `snapshot`-based and `image`-based sandbox creation. If both are set, validation rejects the config as ambiguous.
|
||||
- Reusable leases map to Daytona stop/start semantics. Non-reusable leases are deleted on release.
|
||||
- Reusable leases map to Daytona stop/start semantics. Non-reusable leases are deleted on release. A provider-resolved `target` does not change the identity of an existing sandbox. Release closes the same scoped lease that a later sentinel-verified resume reopens.
|
||||
|
||||
## Local development
|
||||
|
||||
|
|
|
|||
|
|
@ -2551,6 +2551,21 @@ describe("Daytona sandbox provider plugin", () => {
|
|||
expect(mockGet).toHaveBeenCalledTimes(3);
|
||||
});
|
||||
|
||||
it("keeps account credentials and API endpoints isolated for the same sandbox ID", async () => {
|
||||
mockGet.mockImplementation(async () => createMockSandbox({ id: "sandbox-account" }));
|
||||
const params = execParams("sandbox-account");
|
||||
for (const config of [
|
||||
{ apiKey: "account-a", apiUrl: "https://one.daytona.test/api" },
|
||||
{ apiKey: "account-b", apiUrl: "https://one.daytona.test/api" },
|
||||
{ apiKey: "account-a", apiUrl: "https://two.daytona.test/api" },
|
||||
]) {
|
||||
await plugin.definition.onEnvironmentExecute!({
|
||||
...params, config: { ...params.config, ...config },
|
||||
});
|
||||
}
|
||||
expect(mockGet).toHaveBeenCalledTimes(3);
|
||||
});
|
||||
|
||||
it("rejects a queued execute after release teardown closes the lease", async () => {
|
||||
process.env.DAYTONA_API_KEY = "host-key";
|
||||
mockGet.mockImplementation(async () => createMockSandbox({ id: "lease-a" }));
|
||||
|
|
@ -3273,6 +3288,55 @@ describe("Daytona sandbox provider plugin", () => {
|
|||
expect(mockGet).toHaveBeenCalledTimes(2);
|
||||
});
|
||||
|
||||
it("realizes a resumed lease after the provider fills in an unspecified target", async () => {
|
||||
process.env.DAYTONA_API_KEY = "host-key";
|
||||
const sandbox = createMockSandbox({ id: "sandbox-default-target" });
|
||||
mockCreate.mockResolvedValue(sandbox);
|
||||
mockGet.mockResolvedValue(sandbox);
|
||||
const base = { driverKey: "daytona", companyId: "company-1", environmentId: "env-1" };
|
||||
const config = { image: "node:20", timeoutMs: 300000, reuseLease: true };
|
||||
const lease = await plugin.definition.onEnvironmentAcquireLease!({
|
||||
...base, runId: "run-1", agentId: "agent-1", executionWorkspaceId: "workspace-1", config,
|
||||
});
|
||||
// The host materializes provider metadata into later operation config,
|
||||
// but resumes with the environment's original, target-less config.
|
||||
const realizedConfig = { ...config, ...lease.metadata };
|
||||
expect(realizedConfig).toMatchObject({ target: "us" });
|
||||
await plugin.definition.onEnvironmentRealizeWorkspace!({
|
||||
...base, config: realizedConfig, lease,
|
||||
workspace: { remotePath: "/home/daytona/paperclip-workspace" },
|
||||
});
|
||||
expect(mockGet).not.toHaveBeenCalled();
|
||||
await plugin.definition.onEnvironmentReleaseLease!({
|
||||
...base, config: realizedConfig, providerLeaseId: lease.providerLeaseId!,
|
||||
});
|
||||
await expect(plugin.definition.onEnvironmentRealizeWorkspace!({
|
||||
...base, config, lease,
|
||||
workspace: { remotePath: "/home/daytona/paperclip-workspace" },
|
||||
})).rejects.toThrow(/no longer active/);
|
||||
await expect(plugin.definition.onEnvironmentRealizeWorkspace!({
|
||||
...base, config: realizedConfig, lease,
|
||||
workspace: { remotePath: "/home/daytona/paperclip-workspace" },
|
||||
})).rejects.toThrow(/no longer active/);
|
||||
|
||||
sandbox.state = "stopped";
|
||||
const sentinel = lease.metadata!.workspaceSentinel as { token: string };
|
||||
sandbox.process.executeCommand.mockResolvedValueOnce({
|
||||
exitCode: 0, result: JSON.stringify({ token: sentinel.token }),
|
||||
artifacts: { stdout: JSON.stringify({ token: sentinel.token }) },
|
||||
});
|
||||
const resumed = await plugin.definition.onEnvironmentResumeLease!({
|
||||
...base, config, providerLeaseId: lease.providerLeaseId!, leaseMetadata: lease.metadata,
|
||||
});
|
||||
expect(resumed.metadata).toMatchObject({
|
||||
resumedLease: true, workspaceSentinel: { result: "matched" },
|
||||
});
|
||||
await expect(plugin.definition.onEnvironmentRealizeWorkspace!({
|
||||
...base, config: { ...config, ...resumed.metadata }, lease: resumed,
|
||||
workspace: { remotePath: "/home/daytona/paperclip-workspace" },
|
||||
})).resolves.toMatchObject({ cwd: "/home/daytona/paperclip-workspace" });
|
||||
});
|
||||
|
||||
it("realizes the workspace from the acquire-seeded handle without a client.get", async () => {
|
||||
process.env.DAYTONA_API_KEY = "host-key";
|
||||
const sandbox = createMockSandbox({ id: "sandbox-seed" });
|
||||
|
|
|
|||
|
|
@ -965,7 +965,9 @@ function sandboxAccountDiscriminator(config: DaytonaDriverConfig): string {
|
|||
return createHash("sha256")
|
||||
.update(stableStringify({
|
||||
apiUrl: config.apiUrl,
|
||||
target: config.target,
|
||||
// Target is a creation placement hint, not account identity: the SDK
|
||||
// resolves existing sandboxes by ID. Lease metadata fills an omitted
|
||||
// target with the actual region, which must not split admission state.
|
||||
apiKey: resolvedApiKey,
|
||||
}))
|
||||
.digest("hex");
|
||||
|
|
|
|||
|
|
@ -42,6 +42,29 @@ describe("execution truth projection", () => {
|
|||
scheduledRetryAttempt: 12, contextSnapshot: { failureRetriesBeforeWorkspaceWait: 1 } }), undefined, [], undefined, now))
|
||||
.toMatchObject({ label: "Waiting for workspace", phase: "retry_scheduled", attempt: 2, recoveryOwner: null });
|
||||
});
|
||||
it.each([
|
||||
{ status: "resolved", previousRunId: "old", nextRunId: "next", continued: true },
|
||||
{ status: "active", previousRunId: "old", nextRunId: "next", continued: false },
|
||||
{ status: "resolved", previousRunId: "other", nextRunId: "next", continued: false },
|
||||
{ status: "resolved", previousRunId: "old", nextRunId: "old", continued: false },
|
||||
{ status: "resolved", previousRunId: "old", nextRunId: null, continued: false },
|
||||
])("projects the recorded explicit successor without hiding unresolved recovery: $status/$previousRunId/$nextRunId", ({ status, previousRunId, nextRunId, continued }) => {
|
||||
const source = run({ id: "old", status: "failed", errorCode: "adapter_failed" });
|
||||
expect(projectExecution(source, coordinator({ phase: "terminal_failure" }), [], {
|
||||
status,
|
||||
cause: "native_continuation_requires_reconciliation",
|
||||
nextAction: "Inspect the stopped execution.",
|
||||
evidence: {
|
||||
automaticRecovery: { policy: "preserve_without_replay_v1" },
|
||||
explicitUserContinuation: { previousRunId, runId: nextRunId },
|
||||
},
|
||||
}, now)).toMatchObject(continued ? {
|
||||
phase: "completed", label: "Continued in another run", successorRunId: "next",
|
||||
cause: "native_continuation_requires_reconciliation", nextAction: null,
|
||||
} : { phase: "recovery_needed", successorRunId: null });
|
||||
expect(source.status).toBe("failed");
|
||||
});
|
||||
|
||||
it("shows a reconciled continuation as queued until its durable delivery is recorded", () => {
|
||||
const action = {
|
||||
cause: "native_session_retry_exhausted",
|
||||
|
|
|
|||
|
|
@ -184,6 +184,16 @@ export function projectExecution(
|
|||
if (recoveryAction?.status === "resolved" && recoveryAction.evidence?.automaticRecovery) {
|
||||
projection.cause = recoveryAction.cause;
|
||||
projection.nextAction = recoveryAction.nextAction;
|
||||
const continuation = recoveryAction.evidence.explicitUserContinuation as
|
||||
{ previousRunId?: unknown; runId?: unknown } | undefined;
|
||||
const explicitSuccessor = text(continuation?.runId);
|
||||
if (continuation?.previousRunId === run.id && explicitSuccessor && explicitSuccessor !== run.id) {
|
||||
// The admission transaction already recorded the user's successor. Keep
|
||||
// the old failure diagnostic without making it hold the newer attempt.
|
||||
projection.successorRunId = explicitSuccessor;
|
||||
projection.nextAction = null;
|
||||
return set("completed", "Continued in another run");
|
||||
}
|
||||
// Diagnostic projection only: no user decision or replay affordance.
|
||||
return set("recovery_needed", "Stopped");
|
||||
}
|
||||
|
|
|
|||
|
|
@ -9500,6 +9500,77 @@ describe("runnerd provider runtime wiring", () => {
|
|||
);
|
||||
});
|
||||
|
||||
it.each(["fresh", "existing_state", "symlink_parent", "wrong_identity", "connected", "pending_turn", "remote_probe_failed", "backup_present"])(
|
||||
"bootstraps only an untouched provider session in a resumed workspace lease: %s", async (scenario) => {
|
||||
const remoteCwd = join(isolatedStateDirectory, "remote");
|
||||
const runtimeRoot = join(remoteCwd, ".paperclip-runtime", "paperclip-runner");
|
||||
await mkdir(runtimeRoot, { recursive: true });
|
||||
const sessionRoot = join(runtimeRoot, "sessions", createHash("sha256").update(execution.session.normalizedSessionId!).digest("hex"));
|
||||
if (scenario === "existing_state") await mkdir(sessionRoot, { recursive: true });
|
||||
if (scenario === "symlink_parent") await symlink(isolatedStateDirectory, join(runtimeRoot, "sessions"));
|
||||
const remoteExecute = vi.fn(async (command: { command: string; args?: string[] }) => {
|
||||
if (command.args?.[2] === "paperclip-runner-claim-unstarted-session") {
|
||||
let exitCode = 1;
|
||||
if (scenario !== "remote_probe_failed") {
|
||||
try { execFileSync("sh", command.args, { stdio: "pipe" }); exitCode = 0; } catch {}
|
||||
}
|
||||
return { exitCode, timedOut: false, stdout: "", stderr: "" };
|
||||
}
|
||||
if (command.args?.[0] === "--build-metadata") return {
|
||||
exitCode: 0, timedOut: false, stdout: JSON.stringify({
|
||||
schema: "paperclip-runner/runnerd-build-metadata/v1", binaryName: "paperclip-runnerd",
|
||||
packageName: "@paperclipai/paperclip-runner", binaryContractVersion: 2,
|
||||
prpTransportModes: ["listen_ws"],
|
||||
}), stderr: "",
|
||||
};
|
||||
if (command.args?.[0] === "--version") return {
|
||||
exitCode: 0, timedOut: false, stdout: "codex-cli 0.153.4", stderr: "",
|
||||
};
|
||||
if (command.args?.[1]?.includes("base64")) return {
|
||||
exitCode: 1, timedOut: false, stdout: "", stderr: "",
|
||||
};
|
||||
return { exitCode: 0, timedOut: false, stdout: "", stderr: "" };
|
||||
});
|
||||
const backend = await createRunnerdBackend({
|
||||
db: leaseDb(execution), execution, runnerInstanceId: "runner-new-in-retained-workspace",
|
||||
runnerIngressAuthorized: true,
|
||||
runnerExecutionTarget: {
|
||||
kind: "remote", transport: "sandbox", remoteCwd, environmentId: "environment",
|
||||
leaseId: "lease-resumed", providerKey: "daytona",
|
||||
effectiveCapabilities: { runnerWebSocketIngress: true },
|
||||
sandboxLeaseAcquisition: { outcome: "resumed", providerLeaseId: "sandbox-retained" },
|
||||
runner: { execute: remoteExecute },
|
||||
} as never,
|
||||
});
|
||||
expect(backend).toBeDefined();
|
||||
state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!();
|
||||
const options = state.createTransport.mock.calls.at(-1)![0] as RunnerTransportOptions & {
|
||||
prepareExternalRunnerState: () => Promise<void>;
|
||||
};
|
||||
await mkdir(join(options.stateDirectory!, "control-plane"), { recursive: true });
|
||||
await writeFile(join(options.stateDirectory!, "control-plane", "control-plane-state.json"), JSON.stringify({
|
||||
schema: "paperclip.runner.durable.control-plane-state.v1",
|
||||
identity: { ...options.prpIdentity, ...(scenario === "wrong_identity" ? { runId: "other-run" } : {}) },
|
||||
connectionCount: scenario === "connected" ? 1 : 0, committedEvents: [],
|
||||
commands: [{ type: "run.prepare", status: "pending" }, { type: scenario === "pending_turn" ? "turn.start" : "session.open", status: "pending" }],
|
||||
}));
|
||||
if (scenario === "backup_present") {
|
||||
await mkdir(join(options.stateDirectory!, "failover-backups", "current"), { recursive: true });
|
||||
await writeFile(join(options.stateDirectory!, "failover-backups", "current", "manifest.json"), "{}");
|
||||
}
|
||||
if (scenario === "fresh") {
|
||||
await expect(options.prepareExternalRunnerState()).resolves.toBeUndefined();
|
||||
expect(remoteExecute.mock.calls.some(([command]) => command.args?.[1]?.includes("install -d"))).toBe(true);
|
||||
const claimCommand = remoteExecute.mock.calls.find(([command]) => command.args?.[2] === "paperclip-runner-claim-unstarted-session")![0];
|
||||
expect((await lstat(sessionRoot)).mode & 0o777).toBe(0o700);
|
||||
// The exact same claim cannot silently reopen an existing partial root.
|
||||
expect(() => execFileSync("sh", claimCommand.args!, { stdio: "pipe" })).toThrow();
|
||||
} else {
|
||||
await expect(options.prepareExternalRunnerState()).rejects.toThrow("runner_harness_state_mismatch");
|
||||
expect(remoteExecute.mock.calls.some(([command]) => command.args?.[1]?.includes("install -d"))).toBe(false);
|
||||
}
|
||||
});
|
||||
|
||||
it("uses the image's shared Codex without uploading or installing artifacts", async () => {
|
||||
const syncIn = vi.fn(async () => undefined);
|
||||
const remoteExecute = vi.fn(
|
||||
|
|
|
|||
|
|
@ -10702,6 +10702,28 @@ async function createRunnerdBackendWithinSessionClaim(
|
|||
};
|
||||
};
|
||||
|
||||
const claimUntouchedSessionInResumedLease = async (): Promise<boolean> => {
|
||||
if (!remoteCommandRunner || !remoteRuntimeRoot || !remoteSessionRoot) return false;
|
||||
const identity = readRunnerdDurableIdentity(root);
|
||||
if (!durableIdentityMatchesExecution(identity, input.execution) ||
|
||||
identity?.runnerInstanceId !== effectiveRunnerInstanceId ||
|
||||
identity?.environmentLeaseId !== effectiveEnvironmentLeaseId ||
|
||||
!runnerdStateProvesIncompleteBootstrap(root)) return false;
|
||||
// A reusable workspace may have failed before any harness was created.
|
||||
// Claim this exact new session atomically under readable real directories.
|
||||
// Missing files inside an existing session never authorize a fresh start.
|
||||
const probe = await remoteCommandRunner.execute({
|
||||
command: "sh",
|
||||
args: ["-c",
|
||||
'set -eu; umask 077; test -d "$1" && test ! -L "$1" && test -r "$1" && test -x "$1" || exit 1; if test ! -e "$2" && test ! -L "$2"; then mkdir -- "$2"; fi; test -d "$2" && test ! -L "$2" && test -r "$2" && test -x "$2" || exit 1; mkdir -- "$3"',
|
||||
"paperclip-runner-claim-unstarted-session", remoteRuntimeRoot,
|
||||
posix.dirname(remoteSessionRoot), remoteSessionRoot],
|
||||
bypassSession: true,
|
||||
timeoutMs: 10_000,
|
||||
});
|
||||
return probe.exitCode === 0 && !probe.timedOut;
|
||||
};
|
||||
|
||||
const recordInPlaceHarnessReuse = async (
|
||||
providerSessionIdentity: Record<string, unknown>,
|
||||
startedAtMs = Date.now(),
|
||||
|
|
@ -10958,12 +10980,16 @@ async function createRunnerdBackendWithinSessionClaim(
|
|||
!state.runnerState ||
|
||||
!state.providerSessionIdentity
|
||||
) {
|
||||
throw new Error("runner_harness_state_mismatch");
|
||||
if (state.incompleteReason !== "unavailable" || backupAvailable ||
|
||||
!(await claimUntouchedSessionInResumedLease())) {
|
||||
throw new Error("runner_harness_state_mismatch");
|
||||
}
|
||||
} else {
|
||||
await recordInPlaceHarnessReuse(
|
||||
state.providerSessionIdentity,
|
||||
reuseStartedAtMs,
|
||||
);
|
||||
}
|
||||
await recordInPlaceHarnessReuse(
|
||||
state.providerSessionIdentity,
|
||||
reuseStartedAtMs,
|
||||
);
|
||||
} else if (
|
||||
sandboxLeaseAcquisition?.outcome === "replacement"
|
||||
) {
|
||||
|
|
|
|||
|
|
@ -1056,6 +1056,40 @@ describe("TaskChatThread runtime transcript selection", () => {
|
|||
expect(onRetryFailedRun).toHaveBeenCalledWith("native-failed");
|
||||
});
|
||||
|
||||
it.each([false, true])("keeps a later bootstrap failure actionable only after the old recovery has a successor: %s", async (continued) => {
|
||||
const onRetryFailedRun = vi.fn();
|
||||
render(<TaskChatThread comments={[]} onAdd={async () => {}} issueStatus="blocked"
|
||||
onRetryFailedRun={onRetryFailedRun} linkedRuns={[
|
||||
{
|
||||
runId: "old-native", runtimeMode: "native", status: "failed", errorCode: "adapter_failed",
|
||||
agentId: "agent-1", agentName: "Runner", adapterType: "paperclip_runner",
|
||||
createdAt: "2026-08-25T18:00:00.000Z", startedAt: "2026-08-25T18:00:00.000Z",
|
||||
finishedAt: "2026-08-25T18:00:02.000Z",
|
||||
execution: {
|
||||
phase: continued ? "completed" : "recovery_needed", label: continued ? "Continued in another run" : "Stopped",
|
||||
cause: "native_continuation_requires_reconciliation", lastConfirmedActivityAt: null,
|
||||
retryAt: null, attempt: 1, maxAttempts: 3, recoveryOwner: null, nextAction: null,
|
||||
permittedActions: ["inspect_run"], predecessorRunId: null, successorRunId: continued ? "failed-bootstrap" : null,
|
||||
},
|
||||
},
|
||||
{
|
||||
runId: "failed-bootstrap", runtimeMode: "legacy", status: "failed", errorCode: "setup_failed",
|
||||
agentId: "agent-1", agentName: "Runner", adapterType: "paperclip_runner",
|
||||
createdAt: "2026-08-25T18:01:00.000Z", startedAt: "2026-08-25T18:01:00.000Z",
|
||||
finishedAt: "2026-08-25T18:01:02.000Z",
|
||||
},
|
||||
]} />);
|
||||
const retryButtons = Array.from(container.querySelectorAll<HTMLButtonElement>('[data-testid="task-chat-run-failed-try-again"]'));
|
||||
if (!continued) {
|
||||
expect(retryButtons).toHaveLength(0);
|
||||
return;
|
||||
}
|
||||
expect(retryButtons.length).toBeGreaterThan(0);
|
||||
flushSync(() => retryButtons.at(-1)!.click());
|
||||
await Promise.resolve();
|
||||
expect(onRetryFailedRun).toHaveBeenCalledExactlyOnceWith("failed-bootstrap");
|
||||
});
|
||||
|
||||
it("explains a native provider usage limit without exposing its error code", async () => {
|
||||
const onRetryFailedRun = vi.fn();
|
||||
render(
|
||||
|
|
|
|||
Loading…
Reference in New Issue