Test cheap model during agent config check (#8632)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent configuration includes adapter model settings and optional cheap model profiles used by runtime lanes. > - The agent configuration screen already exposes a Test action for adapter environment checks. > - When a cheap model profile is configured, that Test action only exercised the primary model configuration. > - That left users able to save a cheap model profile that had not been validated by the same configuration test flow. > - This pull request makes the Test action probe both the primary model and the configured cheap model. > - The benefit is earlier feedback when the cheap model is unavailable or misconfigured. ## Linked Issues or Issue Description No exact public issue found. Problem description: - What happened: the agent configuration Test action validated the primary adapter model but did not validate an enabled cheap model profile. - Expected behavior: when a cheap model is configured and enabled, the Test action should also test that cheap model using the same environment selection. - Steps to reproduce: configure an agent with a primary model and enabled cheap model profile, then click Test in the agent configuration UI. - Paperclip version/commit: current `master` at PR creation. - Deployment mode: local development UI behavior. Related public context found during GitHub search: - #4881 added cheap model profiles for local adapters. - #6534 is related cheap-primary-model preservation work. - #6956 tracks broader agent configuration settings exposure. GitHub searches performed: - `cheap model config test` - `AgentConfigForm cheap model` - `modelProfiles cheap` - `adapter environment cheap model` ## What Changed - Updated `AgentConfigForm` so the adapter environment Test action runs the primary model check first. - Added a second cheap model check when the cheap profile is enabled and resolves to a model. - Built the cheap test payload from the resolved cheap profile config instead of a model-only override, preserving adapter-default and saved cheap-profile fields. - Merged the individual results into a single labeled result so the UI can show both outcomes together. - Preserved request/API failures so they still surface through the existing error UI instead of becoming synthetic adapter checks. - Added render coverage asserting both primary and cheap test calls, non-model cheap-profile fields, and request failure handling. ## Verification - `git diff origin/master...HEAD | rg -n "(API[_-]?KEY|SECRET|TOKEN|PASSWORD|PRIVATE[_-]?KEY|BEGIN RSA|BEGIN OPENSSH|Bearer [A-Za-z0-9._-]+|ghp_[A-Za-z0-9_]+|sk-[A-Za-z0-9]+|AIza[0-9A-Za-z_-]+|OPENAI_API_KEY|ANTHROPIC_API_KEY)" || true` produced no matches. - `git diff --check origin/master...HEAD` - `pnpm --filter @paperclipai/ui exec vitest run src/components/AgentConfigForm.render.test.tsx --reporter=dot` - `pnpm --filter @paperclipai/ui typecheck` - GitHub PR checks on head `3d9ba15d2` are green, including Build, Typecheck + Release Registry, General tests, serialized server suites, e2e, Canary Dry Run, security scans, and policy/review checks. - Greptile Review passed on head `3d9ba15d2`; all review threads are resolved. ## Risks Low risk. The change is limited to the agent configuration UI test action and its render test. The cheap-model probe now preserves adapter-default and saved cheap-profile fields, matching the runtime merge order more closely. The main residual risk is adapter-specific UI coverage for cheap-profile fields that are stored but not directly editable in this form. ## Model Used OpenAI GPT-5 Codex via the Codex local agent environment, with terminal/tool use for repository inspection, implementation, GitHub CLI operations, and local verification. Exact context window was not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Devin Foley
GitHub
parent
f90ea4dae4
commit
5170a9d35d
|
|
@ -9,9 +9,11 @@ import { TooltipProvider } from "@/components/ui/tooltip";
|
|||
import { AgentConfigForm } from "./AgentConfigForm";
|
||||
|
||||
const mockAgentsApi = vi.hoisted(() => ({
|
||||
adapterModelProfiles: vi.fn(),
|
||||
adapterModels: vi.fn(),
|
||||
detectModel: vi.fn(),
|
||||
list: vi.fn(),
|
||||
testEnvironment: vi.fn(),
|
||||
}));
|
||||
|
||||
const mockEnvironmentsApi = vi.hoisted(() => ({
|
||||
|
|
@ -155,7 +157,11 @@ function makeEnvironment(overrides: Partial<Environment>): Environment {
|
|||
};
|
||||
}
|
||||
|
||||
async function renderForm(environments: Environment[], agentOverrides: Partial<Agent> = {}) {
|
||||
async function renderForm(
|
||||
environments: Environment[],
|
||||
agentOverrides: Partial<Agent> = {},
|
||||
options: { showAdapterTestEnvironmentButton?: boolean } = {},
|
||||
) {
|
||||
mockEnvironmentsApi.list.mockResolvedValue(environments);
|
||||
|
||||
const container = document.createElement("div");
|
||||
|
|
@ -178,7 +184,7 @@ async function renderForm(environments: Environment[], agentOverrides: Partial<A
|
|||
onSave={vi.fn()}
|
||||
hidePromptTemplate
|
||||
showAdapterTypeField={false}
|
||||
showAdapterTestEnvironmentButton={false}
|
||||
showAdapterTestEnvironmentButton={options.showAdapterTestEnvironmentButton ?? false}
|
||||
/>
|
||||
</TooltipProvider>
|
||||
</QueryClientProvider>,
|
||||
|
|
@ -193,9 +199,16 @@ describe("AgentConfigForm environment selector", () => {
|
|||
let roots: Root[] = [];
|
||||
|
||||
beforeEach(() => {
|
||||
mockAgentsApi.adapterModelProfiles.mockResolvedValue([]);
|
||||
mockAgentsApi.adapterModels.mockResolvedValue([]);
|
||||
mockAgentsApi.detectModel.mockResolvedValue(null);
|
||||
mockAgentsApi.list.mockResolvedValue([]);
|
||||
mockAgentsApi.testEnvironment.mockResolvedValue({
|
||||
adapterType: "codex_local",
|
||||
status: "pass",
|
||||
checks: [],
|
||||
testedAt: new Date(0).toISOString(),
|
||||
});
|
||||
mockInstanceSettingsApi.get.mockResolvedValue({ defaultEnvironmentId: null });
|
||||
mockInstanceSettingsApi.getExperimental.mockResolvedValue({ enableEnvironments: true });
|
||||
mockInstanceSettingsApi.getGeneral.mockResolvedValue({ executionMode: "any" });
|
||||
|
|
@ -269,4 +282,83 @@ describe("AgentConfigForm environment selector", () => {
|
|||
expect(selector?.textContent).toContain("Default: Local");
|
||||
expect(selector?.textContent).toContain("Fake Sandbox · sandbox");
|
||||
});
|
||||
|
||||
it("tests both the primary and cheap models when a cheap profile is configured", async () => {
|
||||
const result = await renderForm([
|
||||
makeEnvironment({ id: "local-1", name: "Local", driver: "local" }),
|
||||
], {
|
||||
adapterConfig: { model: "gpt-5.4" },
|
||||
runtimeConfig: {
|
||||
modelProfiles: {
|
||||
cheap: {
|
||||
enabled: true,
|
||||
adapterConfig: {
|
||||
model: "gpt-5.4-mini",
|
||||
baseUrl: "https://cheap-models.example.test",
|
||||
provider: "budget-provider",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
}, {
|
||||
showAdapterTestEnvironmentButton: true,
|
||||
});
|
||||
roots.push(result.root);
|
||||
|
||||
const testButton = Array.from(result.container.querySelectorAll("button")).find(
|
||||
(button) => button.textContent?.trim() === "Test",
|
||||
);
|
||||
expect(testButton).toBeTruthy();
|
||||
|
||||
await act(async () => {
|
||||
testButton?.dispatchEvent(new MouseEvent("click", { bubbles: true }));
|
||||
});
|
||||
await flushReact();
|
||||
|
||||
expect(mockAgentsApi.testEnvironment).toHaveBeenCalledTimes(2);
|
||||
expect(mockAgentsApi.testEnvironment.mock.calls[0]?.[2]).toMatchObject({
|
||||
adapterConfig: expect.objectContaining({ model: "gpt-5.4" }),
|
||||
});
|
||||
expect(mockAgentsApi.testEnvironment.mock.calls[1]?.[2]).toMatchObject({
|
||||
adapterConfig: expect.objectContaining({
|
||||
model: "gpt-5.4-mini",
|
||||
baseUrl: "https://cheap-models.example.test",
|
||||
provider: "budget-provider",
|
||||
}),
|
||||
});
|
||||
});
|
||||
|
||||
it("surfaces request failures instead of converting them into model test checks", async () => {
|
||||
mockAgentsApi.testEnvironment.mockRejectedValueOnce(new Error("Network unavailable"));
|
||||
|
||||
const result = await renderForm([
|
||||
makeEnvironment({ id: "local-1", name: "Local", driver: "local" }),
|
||||
], {
|
||||
adapterConfig: { model: "gpt-5.4" },
|
||||
runtimeConfig: {
|
||||
modelProfiles: {
|
||||
cheap: {
|
||||
enabled: true,
|
||||
adapterConfig: { model: "gpt-5.4-mini" },
|
||||
},
|
||||
},
|
||||
},
|
||||
}, {
|
||||
showAdapterTestEnvironmentButton: true,
|
||||
});
|
||||
roots.push(result.root);
|
||||
|
||||
const testButton = Array.from(result.container.querySelectorAll("button")).find(
|
||||
(button) => button.textContent?.trim() === "Test",
|
||||
);
|
||||
expect(testButton).toBeTruthy();
|
||||
|
||||
await act(async () => {
|
||||
testButton?.dispatchEvent(new MouseEvent("click", { bubbles: true }));
|
||||
});
|
||||
await flushReact();
|
||||
|
||||
expect(mockAgentsApi.testEnvironment).toHaveBeenCalledTimes(1);
|
||||
expect(result.container.textContent).toContain("Network unavailable");
|
||||
});
|
||||
});
|
||||
|
|
|
|||
|
|
@ -516,12 +516,19 @@ export function AgentConfigForm(props: AgentConfigFormProps) {
|
|||
return typeof value === "string" ? value : "";
|
||||
}, [adapterCheapDefault]);
|
||||
|
||||
function buildAdapterConfigForTest(): Record<string, unknown> {
|
||||
function buildAdapterConfigForTest(adapterConfigPatch?: Record<string, unknown>): Record<string, unknown> {
|
||||
if (isCreate) {
|
||||
return uiAdapter.buildAdapterConfig(val!);
|
||||
const next = uiAdapter.buildAdapterConfig(val!);
|
||||
if (adapterConfigPatch) {
|
||||
Object.assign(next, adapterConfigPatch);
|
||||
}
|
||||
return next;
|
||||
}
|
||||
const base = config as Record<string, unknown>;
|
||||
const next = { ...base, ...overlay.adapterConfig };
|
||||
if (adapterConfigPatch) {
|
||||
Object.assign(next, adapterConfigPatch);
|
||||
}
|
||||
if (adapterType === "hermes_local") {
|
||||
const hermesCommand =
|
||||
typeof next.hermesCommand === "string" && next.hermesCommand.length > 0
|
||||
|
|
@ -536,15 +543,123 @@ export function AgentConfigForm(props: AgentConfigFormProps) {
|
|||
return next;
|
||||
}
|
||||
|
||||
function buildCheapAdapterConfigForTest(): Record<string, unknown> {
|
||||
const adapterDefaultConfig = asObject(adapterCheapDefault?.adapterConfig);
|
||||
const createCheapModel = isCreate ? (val!.cheapModel ?? "").trim() : "";
|
||||
const cheapAdapterConfig = isCreate
|
||||
? {
|
||||
...adapterDefaultConfig,
|
||||
...(createCheapModel ? { model: createCheapModel } : {}),
|
||||
}
|
||||
: {
|
||||
...adapterDefaultConfig,
|
||||
...cheapProfileFromAgent.adapterConfig,
|
||||
...asObject(cheapOverlay?.adapterConfig),
|
||||
};
|
||||
return buildAdapterConfigForTest(cheapAdapterConfig);
|
||||
}
|
||||
|
||||
function getCheapModelTestCase(): { model: string; adapterConfig: Record<string, unknown> } | null {
|
||||
if (!currentCheapEnabled) return null;
|
||||
const adapterConfig = buildCheapAdapterConfigForTest();
|
||||
const configModel = typeof adapterConfig.model === "string" ? adapterConfig.model.trim() : "";
|
||||
const model = configModel || currentCheapModel.trim();
|
||||
if (!model) return null;
|
||||
adapterConfig.model = model;
|
||||
return { model, adapterConfig };
|
||||
}
|
||||
|
||||
function prefixEnvironmentTestChecks(
|
||||
result: AdapterEnvironmentTestResult,
|
||||
label: string,
|
||||
model: string | null,
|
||||
): AdapterEnvironmentTestResult {
|
||||
const modelLabel = model ? ` (${model})` : "";
|
||||
return {
|
||||
...result,
|
||||
checks: [
|
||||
{
|
||||
code: `${label.toLowerCase().replace(/[^a-z0-9]+/g, "_")}_test_started`,
|
||||
level: "info",
|
||||
message: `${label} test${modelLabel}`,
|
||||
},
|
||||
...result.checks.map((check) => ({
|
||||
...check,
|
||||
message: `${label} test${modelLabel}: ${check.message}`,
|
||||
})),
|
||||
],
|
||||
};
|
||||
}
|
||||
|
||||
async function runEnvironmentTestCase(
|
||||
label: string,
|
||||
model: string | null,
|
||||
adapterConfig: Record<string, unknown>,
|
||||
environmentId: string | null,
|
||||
): Promise<AdapterEnvironmentTestResult> {
|
||||
const result = await agentsApi.testEnvironment(selectedCompanyId!, adapterType, {
|
||||
adapterConfig,
|
||||
environmentId,
|
||||
});
|
||||
return prefixEnvironmentTestChecks(result, label, model);
|
||||
}
|
||||
|
||||
function mergeEnvironmentTestResults(
|
||||
results: AdapterEnvironmentTestResult[],
|
||||
): AdapterEnvironmentTestResult {
|
||||
const checks = results.flatMap((result) => result.checks);
|
||||
const status = results.some((result) => result.status === "fail")
|
||||
? "fail"
|
||||
: results.some((result) => result.status === "warn")
|
||||
? "warn"
|
||||
: "pass";
|
||||
const testedAt = results[results.length - 1]?.testedAt ?? new Date().toISOString();
|
||||
|
||||
return {
|
||||
adapterType,
|
||||
status,
|
||||
checks,
|
||||
testedAt,
|
||||
};
|
||||
}
|
||||
|
||||
const testEnvironment = useMutation({
|
||||
mutationFn: async () => {
|
||||
if (!selectedCompanyId) {
|
||||
throw new Error("Select a company to test adapter environment");
|
||||
}
|
||||
return agentsApi.testEnvironment(selectedCompanyId, adapterType, {
|
||||
adapterConfig: buildAdapterConfigForTest(),
|
||||
environmentId: currentDefaultEnvironmentId || null,
|
||||
});
|
||||
const primaryModel = currentModelId.trim() || null;
|
||||
const cheapTestCase = getCheapModelTestCase();
|
||||
const environmentId = currentDefaultEnvironmentId || null;
|
||||
const testResults: Array<{ label: string; model: string | null; result: AdapterEnvironmentTestResult }> = [
|
||||
{
|
||||
label: "Primary model",
|
||||
model: primaryModel,
|
||||
result: await runEnvironmentTestCase(
|
||||
"Primary model",
|
||||
primaryModel,
|
||||
buildAdapterConfigForTest(),
|
||||
environmentId,
|
||||
),
|
||||
},
|
||||
];
|
||||
|
||||
if (cheapTestCase) {
|
||||
testResults.push({
|
||||
label: "Cheap model",
|
||||
model: cheapTestCase.model,
|
||||
result: await runEnvironmentTestCase(
|
||||
"Cheap model",
|
||||
cheapTestCase.model,
|
||||
cheapTestCase.adapterConfig,
|
||||
environmentId,
|
||||
),
|
||||
});
|
||||
}
|
||||
|
||||
return testResults.length > 1
|
||||
? mergeEnvironmentTestResults(testResults.map(({ result }) => result))
|
||||
: testResults[0]!.result;
|
||||
},
|
||||
});
|
||||
const [testActionPending, setTestActionPending] = useState(false);
|
||||
|
|
@ -692,9 +807,10 @@ export function AgentConfigForm(props: AgentConfigFormProps) {
|
|||
const cheapProfileFromAgent = useMemo(() => {
|
||||
const profiles = (runtimeConfig.modelProfiles ?? {}) as Record<string, unknown>;
|
||||
const cheap = (profiles.cheap ?? {}) as Record<string, unknown>;
|
||||
const cheapAdapterConfig = (cheap.adapterConfig ?? {}) as Record<string, unknown>;
|
||||
const cheapAdapterConfig = asObject(cheap.adapterConfig);
|
||||
return {
|
||||
enabled: cheap.enabled !== false,
|
||||
adapterConfig: cheapAdapterConfig,
|
||||
model: typeof cheapAdapterConfig.model === "string" ? cheapAdapterConfig.model : "",
|
||||
};
|
||||
}, [runtimeConfig]);
|
||||
|
|
|
|||
Loading…
Reference in New Issue