fix(runner): keep agents running when app connections expire (#12670)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents can receive governed access to connected apps through the runtime MCP gateway. > - A connected app can become unavailable when its sign-in expires or its health state needs attention. > - The native runner treated that optional app state as a fatal runtime setup error. > - One unavailable app could therefore stop all unrelated agent work. > - This pull request removes the fatal dependency and keeps the available app assignment immutable. > - The benefit is that an agent can continue its work while the stream tells the user which app needs reconnection. ## Linked Issues or Issue Description **What happened?** An agent could not start a native run when one assigned app connection was disabled, degraded, failed, or missing its secret. Runtime context creation or MCP delivery threw an error before the agent could do unrelated work. **Expected behavior** The run must continue without the unavailable app. Healthy assigned apps must remain available. The stream must explain which app needs reconnection. A changed assignment must not give a native run new access after its immutable context is captured. **Steps to reproduce** 1. Assign an MCP app connection to a Paperclip Runner agent. 2. Set the connection to a state that needs attention, such as `degraded`. 3. Start a task run for that agent. 4. Observe that native runtime setup fails before the agent starts. **Paperclip version or commit** Reproduced from `ee2a19062`. The branch is rebased on `dda4dff64`. **Deployment mode** Local development from source with embedded Postgres. No matching public issue or open pull request was found in the GitHub search. ## What Changed - Filter unavailable assigned app connections from the immutable native runtime MCP snapshot. - Keep healthy assigned connections and their tools in the snapshot. - Replace the fatal native MCP availability check with an optional stream warning callback. - Withhold MCP delivery when the current assignment digest does not match the captured native context. - Prevent a warning delivery failure from stopping the agent run. - Add regression tests for disabled, degraded, mixed healthy and unavailable, and assignment-drift cases. ## Verification - `pnpm exec vitest run server/src/services/native-runtime/runtime-context.test.ts server/src/__tests__/heartbeat-runtime-mcp-servers.test.ts` passes with 8 tests. - `pnpm -r typecheck` passes. - `pnpm check:token-gates` passes. - `pnpm build` passes. - `pnpm test:run` was attempted. Unrelated workspace runtime and port-exposure tests failed on this macOS host. The same files also failed when run without the changed MCP tests. The changed MCP tests remained green. Clean GitHub CI is the final full-suite check. ## Risks - Low migration risk. This change has no schema or API contract migration. - An unavailable app is absent from the run MCP surface until it is reconnected and a later run captures it again. - Assignment drift fails closed. The agent keeps running, but the changed gateway is not delivered. - This pull request does not auto-block the issue before the agent decides that the app is required. It emits reconnect guidance in the stream. The existing connection-request interaction remains the path for a required app. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex, `gpt-5.6-sol`, with high reasoning, repository tools, code execution, and browser automation. The runtime did not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Dotta
GitHub
parent
dda4dff645
commit
86ebdf842e
|
|
@ -58,7 +58,7 @@ describeEmbeddedPostgres("heartbeat runtime MCP servers", () => {
|
|||
await tempDb?.cleanup();
|
||||
});
|
||||
|
||||
it("provisions one aggregate gateway and filters degraded access without blocking direct adapters", async () => {
|
||||
it("provisions one aggregate gateway and omits unavailable access without blocking any runtime", async () => {
|
||||
process.env.PAPERCLIP_API_URL = "https://paperclip.example.test";
|
||||
const [company] = await db.insert(companies).values({
|
||||
name: `Runtime MCP ${randomUUID()}`,
|
||||
|
|
@ -158,22 +158,35 @@ describeEmbeddedPostgres("heartbeat runtime MCP servers", () => {
|
|||
}
|
||||
expect(JSON.stringify(tokens)).not.toContain(first[0]!.token);
|
||||
|
||||
await db.update(toolConnections)
|
||||
.set({ healthStatus: "degraded", healthMessage: "fixture unavailable" })
|
||||
.where(eq(toolConnections.id, installedConnection!.id));
|
||||
await expect(
|
||||
buildPaperclipRuntimeMcpServers({ db, agent: agent!, runId: randomUUID() }),
|
||||
).resolves.toEqual([]);
|
||||
await expect(
|
||||
buildPaperclipRuntimeMcpServers({
|
||||
db,
|
||||
agent: agent!,
|
||||
runId: randomUUID(),
|
||||
failOnUnavailableAssignedConnection: true,
|
||||
expectedAssignmentDigest: "0".repeat(64),
|
||||
}),
|
||||
).rejects.toThrow(
|
||||
`assigned native MCP connection is unavailable: ${installedConnection!.id}`,
|
||||
);
|
||||
).resolves.toEqual([]);
|
||||
expect(await db.select().from(toolMcpGatewayTokens)).toHaveLength(2);
|
||||
|
||||
await db.update(toolConnections)
|
||||
.set({ healthStatus: "degraded", healthMessage: "fixture unavailable" })
|
||||
.where(eq(toolConnections.id, installedConnection!.id));
|
||||
const unavailableReports: Array<Array<{ id: string; name: string }>> = [];
|
||||
await expect(
|
||||
buildPaperclipRuntimeMcpServers({
|
||||
db,
|
||||
agent: agent!,
|
||||
runId: randomUUID(),
|
||||
expectedAssignmentDigest: first[0]!.connectionId.slice("assignment:".length),
|
||||
onUnavailableAssignedConnections: (connections) => {
|
||||
unavailableReports.push(connections);
|
||||
},
|
||||
}),
|
||||
).resolves.toEqual([]);
|
||||
expect(unavailableReports).toEqual([[
|
||||
{ id: installedConnection!.id, name: installedConnection!.name },
|
||||
]]);
|
||||
expect(await db.select().from(toolMcpGatewayTokens)).toHaveLength(2);
|
||||
await expect(
|
||||
createManagedMcpRunConfig({
|
||||
db,
|
||||
|
|
|
|||
|
|
@ -31,6 +31,7 @@ import {
|
|||
PROVIDER_QUOTA_MONITOR_SERVICE_NAME,
|
||||
envBindingSchema,
|
||||
isEnvironmentDriverSupportedForAdapter,
|
||||
isToolConnectionAttentionHealth,
|
||||
type BillingType,
|
||||
type CostStatus,
|
||||
type EnvironmentLeaseStatus,
|
||||
|
|
@ -3989,7 +3990,10 @@ export async function buildPaperclipRuntimeMcpServers(input: {
|
|||
db: Db;
|
||||
agent: Pick<typeof agents.$inferSelect, "id" | "companyId" | "name">;
|
||||
runId: string;
|
||||
failOnUnavailableAssignedConnection?: boolean;
|
||||
expectedAssignmentDigest?: string | null;
|
||||
onUnavailableAssignedConnections?: (
|
||||
connections: Array<{ id: string; name: string }>,
|
||||
) => void | Promise<void>;
|
||||
}): Promise<AdapterRuntimeMcpServer[]> {
|
||||
const access = toolAccessService(input.db);
|
||||
const effective = await access.getEffectiveProfilesForAgent(
|
||||
|
|
@ -4025,19 +4029,39 @@ export async function buildPaperclipRuntimeMcpServers(input: {
|
|||
permittedConnectionIds.has(connection.id)
|
||||
&& connection.status === "active"
|
||||
&& connection.enabled
|
||||
&& !["degraded", "failed", "error", "missing_secret"].includes(connection.healthStatus)
|
||||
&& !isToolConnectionAttentionHealth(connection.healthStatus)
|
||||
&& (connection.transport === "mcp_remote" || connection.transport === "local_stdio")
|
||||
);
|
||||
const unhealthyConnections = effective.installedConnections.filter((connection) =>
|
||||
permittedConnectionIds.has(connection.id)
|
||||
&& (connection.transport === "mcp_remote" || connection.transport === "local_stdio")
|
||||
&& (!connection.enabled || connection.status !== "active" || ["degraded", "failed", "error", "missing_secret"].includes(connection.healthStatus)),
|
||||
&& (!connection.enabled || connection.status !== "active" || isToolConnectionAttentionHealth(connection.healthStatus)),
|
||||
);
|
||||
if (input.failOnUnavailableAssignedConnection && unhealthyConnections.length) {
|
||||
throw new Error(
|
||||
`assigned native MCP connection is unavailable: ${unhealthyConnections.map((connection) => connection.id).join(", ")}`,
|
||||
);
|
||||
if (unhealthyConnections.length && input.onUnavailableAssignedConnections) {
|
||||
try {
|
||||
await input.onUnavailableAssignedConnections(
|
||||
unhealthyConnections
|
||||
.map(({ id, name }) => ({ id, name }))
|
||||
.sort((a, b) => a.name.localeCompare(b.name)),
|
||||
);
|
||||
} catch (error) {
|
||||
logger.warn(
|
||||
{
|
||||
companyId: input.agent.companyId,
|
||||
agentId: input.agent.id,
|
||||
runId: input.runId,
|
||||
err: error,
|
||||
},
|
||||
"failed to report unavailable runtime MCP connections",
|
||||
);
|
||||
}
|
||||
}
|
||||
const assignedConnectionIds = new Set(
|
||||
assignedConnections.map((connection) => connection.id),
|
||||
);
|
||||
const assignedTools = effective.allowedTools.filter((tool) =>
|
||||
assignedConnectionIds.has(tool.connectionId)
|
||||
);
|
||||
const service = createToolGatewayService(input.db);
|
||||
if (assignedConnections.length === 0) {
|
||||
await service.recordRuntimeMcpDeliveryDiagnostic({
|
||||
|
|
@ -4052,11 +4076,19 @@ export async function buildPaperclipRuntimeMcpServers(input: {
|
|||
version: 1,
|
||||
agentId: input.agent.id,
|
||||
connections: assignedConnections.map((connection) => connection.id).sort(),
|
||||
tools: effective.allowedTools.map((tool) => tool.id).sort(),
|
||||
tools: assignedTools.map((tool) => tool.id).sort(),
|
||||
};
|
||||
const assignmentDigest = createHash("sha256")
|
||||
.update(JSON.stringify(assignment))
|
||||
.digest("hex");
|
||||
// Native runs may lose access after their immutable context is captured, but
|
||||
// they must never gain a new or changed assignment during dispatch.
|
||||
if (
|
||||
input.expectedAssignmentDigest !== undefined
|
||||
&& input.expectedAssignmentDigest !== assignmentDigest
|
||||
) {
|
||||
return [];
|
||||
}
|
||||
const profileKey = `native:${input.agent.id}:${assignmentDigest}`;
|
||||
let [profile] = await input.db
|
||||
.select()
|
||||
|
|
@ -4082,7 +4114,7 @@ export async function buildPaperclipRuntimeMcpServers(input: {
|
|||
applicationId: connection.applicationId,
|
||||
connectionId: connection.id,
|
||||
})),
|
||||
...effective.allowedTools
|
||||
...assignedTools
|
||||
.filter((tool) => !fullConnectionIds.has(tool.connectionId))
|
||||
.map((tool) => ({
|
||||
selectorType: "catalog_entry" as const,
|
||||
|
|
@ -20522,20 +20554,29 @@ export function heartbeatService(
|
|||
if (nativeRuntimeResolution.kind === "native") {
|
||||
if (!nativeExecution || !nativeRunnerInstanceId)
|
||||
throw new Error("native_runtime_selection_not_persisted");
|
||||
const expectedNativeMcpDigest =
|
||||
"runtimeContext" in nativeExecution
|
||||
&& nativeExecution.runtimeContext.mcp.bindingId
|
||||
? nativeExecution.runtimeContext.mcp.digest
|
||||
: null;
|
||||
const nativeMcpServers = await buildPaperclipRuntimeMcpServers({
|
||||
db,
|
||||
agent,
|
||||
runId: run.id,
|
||||
failOnUnavailableAssignedConnection: true,
|
||||
expectedAssignmentDigest: expectedNativeMcpDigest,
|
||||
onUnavailableAssignedConnections: async (connections) => {
|
||||
const names = connections.map((connection) => connection.name).join(", ");
|
||||
await onLog(
|
||||
"stderr",
|
||||
`[paperclip] App connection${connections.length === 1 ? "" : "s"} unavailable: ${names}. Continuing this run without ${connections.length === 1 ? "it" : "them"}; reconnect from Apps to restore access.\n`,
|
||||
);
|
||||
},
|
||||
});
|
||||
if (!("runtimeContext" in nativeExecution) && nativeMcpServers.length) {
|
||||
throw new Error("historical native runs cannot acquire newly assigned MCP access");
|
||||
}
|
||||
if ("runtimeContext" in nativeExecution) {
|
||||
if (nativeMcpServers.length > 1) throw new Error("native MCP realization must produce one aggregate gateway");
|
||||
const server = nativeMcpServers[0] ?? null;
|
||||
const digest = server?.connectionId.startsWith("assignment:") ? server.connectionId.slice("assignment:".length) : null;
|
||||
if (digest !== (nativeExecution.runtimeContext.mcp.bindingId ? nativeExecution.runtimeContext.mcp.digest : null)) {
|
||||
if (digest && digest !== expectedNativeMcpDigest) {
|
||||
throw new Error("native MCP assignment digest mismatch");
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -83,6 +83,102 @@ afterEach(async () => {
|
|||
});
|
||||
|
||||
describe("buildNativeRuntimeContext", () => {
|
||||
it.each(["disabled", "degraded"] as const)(
|
||||
"omits an unavailable native MCP connection when it is %s without aborting runtime context creation",
|
||||
async (unavailableState) => {
|
||||
serviceMocks.exportFiles.mockResolvedValue({
|
||||
entryFile: "AGENTS.md",
|
||||
files: { "AGENTS.md": "Continue work without unavailable apps.\n" },
|
||||
});
|
||||
serviceMocks.getEffectiveProfilesForAgent.mockResolvedValue({
|
||||
agentId: "agent-1",
|
||||
profiles: [],
|
||||
entries: [{ effect: "include", connectionId: "connection-1" }],
|
||||
bindings: [],
|
||||
allowedTools: [{ id: "tool-1", connectionId: "connection-1" }],
|
||||
allowedToolNames: ["issues.read"],
|
||||
installedConnections: [{
|
||||
id: "connection-1",
|
||||
transport: "mcp_remote",
|
||||
enabled: unavailableState !== "disabled",
|
||||
status: unavailableState === "disabled" ? "disabled" : "active",
|
||||
healthStatus: unavailableState === "degraded" ? "degraded" : "healthy",
|
||||
}],
|
||||
});
|
||||
|
||||
const context = await buildNativeRuntimeContext({
|
||||
db: {} as Db,
|
||||
agent: {
|
||||
id: "agent-1",
|
||||
companyId: "company-1",
|
||||
name: "Reviewer",
|
||||
adapterType: "paperclip_runner",
|
||||
adapterConfig: {},
|
||||
},
|
||||
runId: "run-1",
|
||||
runtimeConfig: {},
|
||||
runtimeSkillEntries: [],
|
||||
});
|
||||
|
||||
expect(context.mcp.bindingId).toBeNull();
|
||||
expect(context.mcp.assignmentSetId).toMatch(/^sha256:[a-f0-9]{64}$/);
|
||||
},
|
||||
);
|
||||
|
||||
it("keeps healthy native MCP connections when another assigned connection is unavailable", async () => {
|
||||
serviceMocks.exportFiles.mockResolvedValue({
|
||||
entryFile: "AGENTS.md",
|
||||
files: { "AGENTS.md": "Continue work with the apps that are available.\n" },
|
||||
});
|
||||
serviceMocks.getEffectiveProfilesForAgent.mockResolvedValue({
|
||||
agentId: "agent-1",
|
||||
profiles: [],
|
||||
entries: [
|
||||
{ effect: "include", connectionId: "connection-expired" },
|
||||
{ effect: "include", connectionId: "connection-healthy" },
|
||||
],
|
||||
bindings: [],
|
||||
allowedTools: [
|
||||
{ id: "tool-expired", connectionId: "connection-expired" },
|
||||
{ id: "tool-healthy", connectionId: "connection-healthy" },
|
||||
],
|
||||
allowedToolNames: ["expired.read", "healthy.read"],
|
||||
installedConnections: [
|
||||
{
|
||||
id: "connection-expired",
|
||||
transport: "mcp_remote",
|
||||
enabled: true,
|
||||
status: "active",
|
||||
healthStatus: "degraded",
|
||||
},
|
||||
{
|
||||
id: "connection-healthy",
|
||||
transport: "mcp_remote",
|
||||
enabled: true,
|
||||
status: "active",
|
||||
healthStatus: "healthy",
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
const context = await buildNativeRuntimeContext({
|
||||
db: {} as Db,
|
||||
agent: {
|
||||
id: "agent-1",
|
||||
companyId: "company-1",
|
||||
name: "Reviewer",
|
||||
adapterType: "paperclip_runner",
|
||||
adapterConfig: {},
|
||||
},
|
||||
runId: "run-1",
|
||||
runtimeConfig: {},
|
||||
runtimeSkillEntries: [],
|
||||
});
|
||||
|
||||
expect(context.mcp.bindingId).toBe("native-mcp:run-1");
|
||||
expect(context.mcp.assignmentSetId).toMatch(/^sha256:[a-f0-9]{64}$/);
|
||||
});
|
||||
|
||||
it("materializes every instruction and selected-skill file as immutable, content-addressed context", async () => {
|
||||
serviceMocks.exportFiles.mockResolvedValue({
|
||||
entryFile: "AGENTS.md",
|
||||
|
|
|
|||
|
|
@ -3,6 +3,7 @@ import fs from "node:fs/promises";
|
|||
import path from "node:path";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import type { PaperclipSkillEntry } from "@paperclipai/adapter-utils/server-utils";
|
||||
import { isToolConnectionAttentionHealth } from "@paperclipai/shared";
|
||||
import {
|
||||
PAPERCLIP_OPERATIONAL_SKILL_KEY,
|
||||
resolvePaperclipDesiredSkillNames,
|
||||
|
|
@ -165,17 +166,23 @@ async function materializeSelectedSkills(runtimeConfig: Record<string, unknown>,
|
|||
export async function resolveNativeRuntimeMcpSnapshot(input: { db: Db; agent: Pick<RuntimeAgent, "id" | "companyId">; runId: string }) {
|
||||
const effective = await toolAccessService(input.db).getEffectiveProfilesForAgent(input.agent.companyId, input.agent.id);
|
||||
const permitted = new Set([...effective.entries.filter((entry) => entry.effect === "include" && entry.connectionId).map((entry) => entry.connectionId!), ...effective.allowedTools.map((tool) => tool.connectionId)]);
|
||||
const unhealthy = effective.installedConnections.filter((connection) =>
|
||||
// App access is optional runtime context. Keep usable assignments pinned, but
|
||||
// do not stop unrelated work because an assigned app needs attention.
|
||||
const availableConnectionIds = new Set(effective.installedConnections.filter((connection) =>
|
||||
permitted.has(connection.id)
|
||||
&& connection.status === "active"
|
||||
&& connection.enabled
|
||||
&& !isToolConnectionAttentionHealth(connection.healthStatus)
|
||||
&& ["mcp_remote", "local_stdio"].includes(connection.transport)
|
||||
&& (!connection.enabled || connection.status !== "active" || ["degraded", "failed", "error", "missing_secret"].includes(connection.healthStatus)),
|
||||
);
|
||||
if (unhealthy.length) throw new Error(`assigned native MCP connection is unavailable: ${unhealthy.map((connection) => connection.id).join(", ")}`);
|
||||
).map((connection) => connection.id));
|
||||
const assignment = {
|
||||
version: 1,
|
||||
agentId: input.agent.id,
|
||||
connections: effective.installedConnections.filter((connection) => permitted.has(connection.id) && connection.status === "active" && connection.enabled && ["mcp_remote", "local_stdio"].includes(connection.transport)).map((connection) => connection.id).sort(),
|
||||
tools: effective.allowedTools.map((tool) => tool.id).sort(),
|
||||
connections: [...availableConnectionIds].sort(),
|
||||
tools: effective.allowedTools
|
||||
.filter((tool) => availableConnectionIds.has(tool.connectionId))
|
||||
.map((tool) => tool.id)
|
||||
.sort(),
|
||||
};
|
||||
const assignmentDigest = sha256(JSON.stringify(assignment));
|
||||
return { assignmentSetId: `sha256:${assignmentDigest}`, digest: assignmentDigest, bindingId: assignment.connections.length ? `native-mcp:${input.runId}` : null };
|
||||
|
|
|
|||
Loading…
Reference in New Issue