test(runner): add question adapter conformance (#12409)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The runner maps provider questions to one versioned Paperclip contract. > - Codex and ACPX now perform that mapping through separate adapters. > - Separate adapter tests do not prove that both paths preserve the same user-visible form. > - Fixture validation must also match production behavior for optional answers and unsupported patterns. > - This pull request adds shared fixtures, validation, and cross-adapter conformance checks. > - The benefit is a reusable question contract for later providers without enabling a new runtime. ## Linked Issues or Issue Description Refs #12408 This pull request builds on the Codex ACPX question bridge merged in #12408. It adds fixture and generator checks for the existing Codex and ACPX question adapters. It does not add another provider or production execution path. ## What Changed - Add a canonical ACPX form fixture and native response. Mark the equivalent Codex fixture field as required. - Add shared validation for question IDs, option IDs, answer modes, required answers, text bounds, numeric bounds, and response shapes. - Evaluate fixture-only regular expressions in a bounded child process. Reject patterns that cannot finish safely. - Validate ACPX fixtures against a manifest-side mirror of the production form projection. Reject free-text ACPX patterns, but ignore patterns on enumerated option fields. - Accept explicit empty optional answers and omit them from the projected ACPX response, which matches the production parser. - Validate every question fixture during manifest generation and regenerate the checked-in manifest. - Add cross-adapter tests that compare user-visible presentation while preserving provider-owned IDs and provider-specific response conversion. - Add negative regressions for malformed forms, invalid responses, unsafe patterns, special property names, and projection drift. ## Verification - Replay base: `4fe3189f0256873a359d2d53c209076919fd1c3b` (`master` after #12408 merged). - Exact replay head: `0532e7dfbb5a246033ffeef55a0c0013fdab07f1`. - Stable patch ID for the intended seven-file delta: `28154d86b2c37e0e8d442419e26703584852f67e`. - The intended pull request delta contains exactly these seven files: - `packages/paperclip-runner/protocol/fixtures/questions/acpx.json` - `packages/paperclip-runner/protocol/fixtures/questions/codex.json` - `packages/paperclip-runner/protocol/manifest.json` - `packages/paperclip-runner/scripts/generate-protocol-manifest.mjs` - `packages/paperclip-runner/scripts/protocol-contract.mjs` - `packages/paperclip-runner/src/contracts/question-adapter-conformance.test.ts` - `packages/paperclip-runner/test/protocol-contract.test.mjs` - The intended combined delta is 1,211 additions and 20 deletions. - This change does not add a dependency, lockfile update, migration, workflow, server route, UI change, documentation file, or production runtime change. - GitHub Actions run `33352004952` passed the complete matrix on retry at the unchanged exact head, including protocol/package verification, build, typecheck/release-registry, general and serialized server suites, canary, and all e2e shards. - Superagent, Socket, Snyk, contributor-trust, policy, and PR-review checks pass on the exact replay head. - Greptile reviewed the exact replay head at 5/5 with no blocking finding and zero unresolved review threads. - No local test result is claimed. GitHub Actions is the authoritative verification environment for the replayed revision. ## Risks This change has low runtime risk because it changes fixtures, generator validation, generated metadata, and tests only. Fixture pattern checks run in a child process with a one-second timeout and a bounded output buffer. The ACPX gate intentionally rejects free-text patterns because the production adapter has no bounded expression engine. It intentionally permits an explicit empty optional answer because production omits that answer from the native response. A validation mismatch can block manifest generation, but it cannot change server selection, direct adapters, or task-page behavior. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex with GPT-5.6, extended reasoning, repository tool use, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge
This commit is contained in:
Dotta
GitHub
parent
4fe3189f02
commit
b93ad538b6
|
|
@ -0,0 +1,75 @@
|
|||
{
|
||||
"schema": "paperclip.question_adapter_fixture.v1",
|
||||
"adapter": "acpx",
|
||||
"nativeRequest": {
|
||||
"method": "elicitation/create",
|
||||
"params": {
|
||||
"mode": "form",
|
||||
"message": "Deployment input",
|
||||
"requestedSchema": {
|
||||
"type": "object",
|
||||
"title": "Deployment input",
|
||||
"required": ["environment"],
|
||||
"properties": {
|
||||
"environment": {
|
||||
"type": "string",
|
||||
"title": "Environment",
|
||||
"description": "Where should we deploy?",
|
||||
"oneOf": [
|
||||
{
|
||||
"const": "staging",
|
||||
"title": "Staging",
|
||||
"description": "Deploy to staging first."
|
||||
},
|
||||
{
|
||||
"const": "production",
|
||||
"title": "Production",
|
||||
"description": "Deploy directly to production."
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"canonicalQuestionSet": {
|
||||
"schema": "paperclip.question_set.v1",
|
||||
"title": "Deployment input",
|
||||
"submitLabel": "Submit answers",
|
||||
"questions": [
|
||||
{
|
||||
"id": "field-1-environment-ba5285161ba6",
|
||||
"header": "Environment",
|
||||
"prompt": "Where should we deploy?",
|
||||
"required": true,
|
||||
"answerMode": "single_select",
|
||||
"options": [
|
||||
{
|
||||
"id": "option-1",
|
||||
"label": "Staging",
|
||||
"description": "Deploy to staging first."
|
||||
},
|
||||
{
|
||||
"id": "option-2",
|
||||
"label": "Production",
|
||||
"description": "Deploy directly to production."
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
"canonicalResponse": {
|
||||
"schema": "paperclip.question_response.v1",
|
||||
"answers": {
|
||||
"field-1-environment-ba5285161ba6": {
|
||||
"selectedOptionIds": ["option-1"]
|
||||
}
|
||||
}
|
||||
},
|
||||
"nativeResponse": {
|
||||
"action": "accept",
|
||||
"content": {
|
||||
"environment": "staging"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -9,6 +9,7 @@
|
|||
"id": "environment",
|
||||
"header": "Environment",
|
||||
"question": "Where should we deploy?",
|
||||
"required": true,
|
||||
"options": [
|
||||
{ "label": "Staging", "description": "Deploy to staging first." },
|
||||
{ "label": "Production", "description": "Deploy directly to production." }
|
||||
|
|
|
|||
|
|
@ -175,9 +175,15 @@
|
|||
"expectation": "accept",
|
||||
"compatibilityCase": "canonical"
|
||||
},
|
||||
{
|
||||
"path": "fixtures/questions/acpx.json",
|
||||
"sha256": "6f1b65d5168b39f4bc821e778901bf077b1df10495f4bc1217b515eca4a339e4",
|
||||
"expectation": "accept",
|
||||
"compatibilityCase": "acpx-structured-input"
|
||||
},
|
||||
{
|
||||
"path": "fixtures/questions/codex.json",
|
||||
"sha256": "ad673b6f9d70b74b53656e622d15fad1a66430f39d62b9ef71479d194a198bd6",
|
||||
"sha256": "a02a862b008d8327b4665bbde8ae3b73ad1f8dca7891072b0839b0e84b5920fa",
|
||||
"expectation": "accept",
|
||||
"compatibilityCase": "codex-structured-input"
|
||||
},
|
||||
|
|
|
|||
|
|
@ -5,8 +5,10 @@ import { fileURLToPath } from "node:url";
|
|||
import {
|
||||
SUPPORTED_FIXTURE_VERSION,
|
||||
SUPPORTED_PROTOCOL_VERSION,
|
||||
assertAcpxQuestionFixture,
|
||||
assertCodexQuestionFixture,
|
||||
assertConformanceFixturePair,
|
||||
assertQuestionAdapterFixture,
|
||||
assertReplayFixtureCompatibility,
|
||||
assertSchemaInstance,
|
||||
compileProtocolValidators,
|
||||
|
|
@ -64,14 +66,19 @@ export async function buildProtocolManifest() {
|
|||
compatibilityCase = "additive-optional-fields";
|
||||
}
|
||||
}
|
||||
} else if (relativePath === "fixtures/questions/codex.json") {
|
||||
assertCodexQuestionFixture(value);
|
||||
} else if (relativePath.startsWith("fixtures/questions/")) {
|
||||
assertSchemaInstance(
|
||||
validators.questionAdapterFixture,
|
||||
value,
|
||||
relativePath,
|
||||
);
|
||||
compatibilityCase = "codex-structured-input";
|
||||
assertQuestionAdapterFixture(value);
|
||||
if (relativePath === "fixtures/questions/codex.json") {
|
||||
assertCodexQuestionFixture(value);
|
||||
} else if (relativePath === "fixtures/questions/acpx.json") {
|
||||
assertAcpxQuestionFixture(value);
|
||||
}
|
||||
compatibilityCase = `${value.adapter}-structured-input`;
|
||||
} else if (relativePath === "fixtures/conformance-minimal-run.json") {
|
||||
assertSchemaInstance(validators.conformanceFixture, value, relativePath);
|
||||
compatibilityCase = "cross-language-input";
|
||||
|
|
|
|||
|
|
@ -1,3 +1,4 @@
|
|||
import { spawnSync } from "node:child_process";
|
||||
import { createHash } from "node:crypto";
|
||||
import { readdir, readFile } from "node:fs/promises";
|
||||
import { relative, resolve, sep } from "node:path";
|
||||
|
|
@ -180,48 +181,521 @@ export function assertReplayFixtureCompatibility(fixture) {
|
|||
return fixture;
|
||||
}
|
||||
|
||||
export function assertCodexQuestionFixture(fixture) {
|
||||
export function assertQuestionAdapterFixture(fixture) {
|
||||
requireSchema(fixture, "paperclip.question_adapter_fixture.v1", "question fixture");
|
||||
if (fixture.adapter !== "codex") throw contractError("unsupported_provider", String(fixture.adapter));
|
||||
requireSchema(fixture.canonicalQuestionSet, "paperclip.question_set.v1", "canonicalQuestionSet");
|
||||
requireSchema(fixture.canonicalResponse, "paperclip.question_response.v1", "canonicalResponse");
|
||||
if (fixture.nativeRequest?.method !== "item/tool/requestUserInput") {
|
||||
throw contractError("invalid_codex_question_fixture", "native request method");
|
||||
}
|
||||
|
||||
const questions = fixture.canonicalQuestionSet.questions;
|
||||
if (!Array.isArray(questions) || questions.length === 0) {
|
||||
throw contractError("invalid_codex_question_fixture", "questions must be non-empty");
|
||||
throw contractError("invalid_question_adapter_fixture", "questions must be non-empty");
|
||||
}
|
||||
const questionIds = new Set();
|
||||
const questionsById = new Map();
|
||||
const optionIdsByQuestion = new Map();
|
||||
for (const question of questions) {
|
||||
if (typeof question.id !== "string" || question.id.length === 0 || questionIds.has(question.id)) {
|
||||
throw contractError("invalid_codex_question_fixture", "question IDs must be unique");
|
||||
if (typeof question.id !== "string" || question.id.length === 0 || questionsById.has(question.id)) {
|
||||
throw contractError("invalid_question_adapter_fixture", "question IDs must be unique");
|
||||
}
|
||||
questionIds.add(question.id);
|
||||
questionsById.set(question.id, question);
|
||||
const optionIds = new Set();
|
||||
for (const option of question.options ?? []) {
|
||||
if (typeof option.id !== "string" || option.id.length === 0 || optionIds.has(option.id)) {
|
||||
throw contractError("invalid_codex_question_fixture", `option IDs for ${question.id} must be unique`);
|
||||
throw contractError("invalid_question_adapter_fixture", `option IDs for ${question.id} must be unique`);
|
||||
}
|
||||
optionIds.add(option.id);
|
||||
}
|
||||
optionIdsByQuestion.set(question.id, optionIds);
|
||||
}
|
||||
for (const [answerId, answer] of Object.entries(fixture.canonicalResponse.answers ?? {})) {
|
||||
if (!questionIds.has(answerId)) {
|
||||
throw contractError("invalid_codex_question_fixture", `answer has unknown question ID ${answerId}`);
|
||||
const validation = question.textValidation;
|
||||
if (
|
||||
validation?.minLength !== undefined
|
||||
&& validation.maxLength !== undefined
|
||||
&& validation.minLength > validation.maxLength
|
||||
) {
|
||||
throw contractError("invalid_question_adapter_fixture", `text validation for ${question.id} has minLength greater than maxLength`);
|
||||
}
|
||||
for (const optionId of answer.selectedOptionIds ?? []) {
|
||||
if (
|
||||
validation?.minimum !== undefined
|
||||
&& validation.maximum !== undefined
|
||||
&& validation.minimum > validation.maximum
|
||||
) {
|
||||
throw contractError("invalid_question_adapter_fixture", `text validation for ${question.id} has minimum greater than maximum`);
|
||||
}
|
||||
if (validation?.pattern !== undefined) {
|
||||
try {
|
||||
new RegExp(validation.pattern);
|
||||
} catch {
|
||||
throw contractError("invalid_question_adapter_fixture", `text validation pattern for ${question.id} is invalid`);
|
||||
}
|
||||
}
|
||||
}
|
||||
const answers = fixture.canonicalResponse.answers ?? {};
|
||||
for (const [answerId, answer] of Object.entries(answers)) {
|
||||
if (!questionsById.has(answerId)) {
|
||||
throw contractError("invalid_question_adapter_fixture", `answer has unknown question ID ${answerId}`);
|
||||
}
|
||||
if (answer === null || typeof answer !== "object" || Array.isArray(answer)) {
|
||||
throw contractError("invalid_question_adapter_fixture", `answer for ${answerId} must be an object`);
|
||||
}
|
||||
const selectedOptionIds = answer.selectedOptionIds ?? [];
|
||||
if (!Array.isArray(selectedOptionIds) || selectedOptionIds.some((id) => typeof id !== "string")) {
|
||||
throw contractError("invalid_question_adapter_fixture", `answer for ${answerId} has invalid option IDs`);
|
||||
}
|
||||
if (new Set(selectedOptionIds).size !== selectedOptionIds.length) {
|
||||
throw contractError("invalid_question_adapter_fixture", `answer for ${answerId} repeats an option ID`);
|
||||
}
|
||||
for (const optionId of selectedOptionIds) {
|
||||
if (!optionIdsByQuestion.get(answerId)?.has(optionId)) {
|
||||
throw contractError("invalid_codex_question_fixture", `answer has unknown option ID ${optionId}`);
|
||||
throw contractError("invalid_question_adapter_fixture", `answer has unknown option ID ${optionId}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
for (const question of questions) {
|
||||
const answer = answers[question.id];
|
||||
if (answer === undefined) {
|
||||
if (question.required) {
|
||||
throw contractError("invalid_question_adapter_fixture", `answer for required question ${question.id} is missing`);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
const selectedOptionIds = answer.selectedOptionIds ?? [];
|
||||
const text = answer.text;
|
||||
const customText = answer.customText;
|
||||
if (question.answerMode === "text") {
|
||||
if (selectedOptionIds.length > 0 || customText !== undefined) {
|
||||
throw contractError("invalid_question_adapter_fixture", `text answer for ${question.id} carries select-only fields`);
|
||||
}
|
||||
} else {
|
||||
if (text !== undefined) {
|
||||
throw contractError("invalid_question_adapter_fixture", `select answer for ${question.id} carries text`);
|
||||
}
|
||||
if (question.answerMode === "single_select" && selectedOptionIds.length > 1) {
|
||||
throw contractError("invalid_question_adapter_fixture", `single-select answer for ${question.id} chooses more than one option`);
|
||||
}
|
||||
if (customText !== undefined && question.customAnswer?.enabled !== true) {
|
||||
throw contractError("invalid_question_adapter_fixture", `custom answer for ${question.id} is not enabled`);
|
||||
}
|
||||
if (
|
||||
question.answerMode === "single_select"
|
||||
&& typeof customText === "string"
|
||||
&& customText.trim().length > 0
|
||||
&& selectedOptionIds.length > 0
|
||||
) {
|
||||
throw contractError("invalid_question_adapter_fixture", `single-select answer for ${question.id} mixes option and custom values`);
|
||||
}
|
||||
}
|
||||
const hasValue = fixtureAnswerHasValue(answer);
|
||||
if (question.required && !hasValue) {
|
||||
throw contractError("invalid_question_adapter_fixture", `answer for required question ${question.id} is empty`);
|
||||
}
|
||||
const boundedText = question.answerMode === "text" ? text : customText;
|
||||
if (boundedText !== undefined) {
|
||||
const validation = question.textValidation;
|
||||
if (validation?.minLength !== undefined && boundedText.length < validation.minLength) {
|
||||
throw contractError("invalid_question_adapter_fixture", `answer for ${question.id} must contain at least ${validation.minLength} characters`);
|
||||
}
|
||||
if (validation?.maxLength !== undefined && boundedText.length > validation.maxLength) {
|
||||
throw contractError("invalid_question_adapter_fixture", `answer for ${question.id} must contain at most ${validation.maxLength} characters`);
|
||||
}
|
||||
if (validation?.pattern !== undefined) {
|
||||
let pattern;
|
||||
try {
|
||||
pattern = new RegExp(validation.pattern);
|
||||
} catch {
|
||||
throw contractError("invalid_question_adapter_fixture", `text validation pattern for ${question.id} is invalid`);
|
||||
}
|
||||
if (!testFixturePatternWithDeadline(pattern.source, boundedText, question.id)) {
|
||||
throw contractError("invalid_question_adapter_fixture", `answer for ${question.id} does not match the required format`);
|
||||
}
|
||||
}
|
||||
if (validation?.inputType === "number" || validation?.inputType === "integer") {
|
||||
const numeric = Number(boundedText);
|
||||
if (!Number.isFinite(numeric) || (validation.inputType === "integer" && !Number.isInteger(numeric))) {
|
||||
throw contractError("invalid_question_adapter_fixture", `answer for ${question.id} must be a valid ${validation.inputType}`);
|
||||
}
|
||||
if (validation.minimum !== undefined && numeric < validation.minimum) {
|
||||
throw contractError("invalid_question_adapter_fixture", `answer for ${question.id} must be at least ${validation.minimum}`);
|
||||
}
|
||||
if (validation.maximum !== undefined && numeric > validation.maximum) {
|
||||
throw contractError("invalid_question_adapter_fixture", `answer for ${question.id} must be at most ${validation.maximum}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return fixture;
|
||||
}
|
||||
|
||||
export function assertCodexQuestionFixture(fixture) {
|
||||
assertQuestionAdapterFixture(fixture);
|
||||
if (fixture.adapter !== "codex") throw contractError("unsupported_provider", String(fixture.adapter));
|
||||
if (fixture.nativeRequest?.method !== "item/tool/requestUserInput") {
|
||||
throw contractError("invalid_codex_question_fixture", "native request method");
|
||||
}
|
||||
return fixture;
|
||||
}
|
||||
|
||||
export function assertAcpxQuestionFixture(fixture) {
|
||||
assertQuestionAdapterFixture(fixture);
|
||||
if (fixture.adapter !== "acpx") {
|
||||
throw contractError("unsupported_provider", String(fixture.adapter));
|
||||
}
|
||||
const request = fixture.nativeRequest;
|
||||
if (!isPlainRecord(request) || request.method !== "elicitation/create") {
|
||||
throw contractError("invalid_acpx_question_fixture", "native request method");
|
||||
}
|
||||
const params = request.params;
|
||||
const requestedSchema = isPlainRecord(params) ? params.requestedSchema : null;
|
||||
if (
|
||||
!isPlainRecord(params)
|
||||
|| params.mode !== "form"
|
||||
|| !isPlainRecord(requestedSchema)
|
||||
|| !isPlainRecord(requestedSchema.properties)
|
||||
) {
|
||||
throw contractError("invalid_acpx_question_fixture", "native form request");
|
||||
}
|
||||
const properties = Object.entries(requestedSchema.properties);
|
||||
if (properties.length === 0 || properties.length > 64) {
|
||||
throw contractError("invalid_acpx_question_fixture", "native form property count");
|
||||
}
|
||||
const required = normalizedAcpxRequired(requestedSchema);
|
||||
for (const [name, property] of properties) {
|
||||
assertAcpxProperty(name, property);
|
||||
if (
|
||||
isPlainRecord(property)
|
||||
&& property.type === "string"
|
||||
&& acpxEnumValues(property, "oneOf").length === 0
|
||||
&& optionalFixtureText(property.pattern) !== undefined
|
||||
) {
|
||||
throw contractError(
|
||||
"invalid_acpx_question_fixture",
|
||||
`native property ${name} uses an unsupported pattern`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
const response = fixture.nativeResponse;
|
||||
if (
|
||||
!isPlainRecord(response)
|
||||
|| response.action !== "accept"
|
||||
|| !isPlainRecord(response.content)
|
||||
) {
|
||||
throw contractError("invalid_acpx_question_fixture", "native accept response");
|
||||
}
|
||||
for (const [name] of properties) {
|
||||
if (required.has(name) && !Object.hasOwn(response.content, name)) {
|
||||
throw contractError("invalid_acpx_question_fixture", `native response omits required property ${name}`);
|
||||
}
|
||||
}
|
||||
for (const [name, value] of Object.entries(response.content)) {
|
||||
const property = requestedSchema.properties[name];
|
||||
if (!isPlainRecord(property)) {
|
||||
throw contractError("invalid_acpx_question_fixture", `native response has unknown property ${name}`);
|
||||
}
|
||||
assertAcpxPropertyValue(name, property, value);
|
||||
}
|
||||
const projected = projectAcpxFixture(params, fixture.canonicalResponse);
|
||||
if (canonicalFixtureJson(projected.questionSet) !== canonicalFixtureJson(fixture.canonicalQuestionSet)) {
|
||||
throw contractError("invalid_acpx_question_fixture", "native and canonical question sets differ");
|
||||
}
|
||||
if (canonicalFixtureJson(projected.nativeResponse) !== canonicalFixtureJson(fixture.nativeResponse)) {
|
||||
throw contractError("invalid_acpx_question_fixture", "canonical and native responses differ");
|
||||
}
|
||||
return fixture;
|
||||
}
|
||||
|
||||
function projectAcpxFixture(params, canonicalResponse) {
|
||||
const schema = params.requestedSchema;
|
||||
const required = normalizedAcpxRequired(schema);
|
||||
const bindings = Object.entries(schema.properties).map(([name, property], index) =>
|
||||
projectAcpxProperty(name, property, index, required.has(name))
|
||||
);
|
||||
const title = optionalFixtureText(schema.title) ?? "Additional information needed";
|
||||
const descriptions = [
|
||||
optionalFixtureText(params.message),
|
||||
optionalFixtureText(schema.description),
|
||||
].filter((value, index, all) => value !== undefined && value !== title && all.indexOf(value) === index);
|
||||
const questionSet = {
|
||||
schema: "paperclip.question_set.v1",
|
||||
title,
|
||||
...(descriptions.length > 0 ? { description: descriptions.join("\n\n") } : {}),
|
||||
submitLabel: "Submit answers",
|
||||
questions: bindings.map((binding) => binding.question),
|
||||
};
|
||||
const content = {};
|
||||
for (const binding of bindings) {
|
||||
const answer = canonicalResponse.answers?.[binding.question.id];
|
||||
// The production parser accepts explicit empty optional answers and omits
|
||||
// them from its normalized response. Mirror that before projecting ACP
|
||||
// content so the fixture gate certifies the same wire behavior.
|
||||
if (!answer || !fixtureAnswerHasValue(answer)) continue;
|
||||
if (binding.type === "string" && binding.question.answerMode === "text") {
|
||||
if (answer.text !== undefined) defineAcpxResponseProperty(content, binding.name, answer.text);
|
||||
} else if (binding.type === "number" || binding.type === "integer") {
|
||||
if (answer.text !== undefined) {
|
||||
defineAcpxResponseProperty(content, binding.name, Number(answer.text));
|
||||
}
|
||||
} else if (binding.type === "boolean") {
|
||||
const selected = answer.selectedOptionIds?.[0];
|
||||
if (selected !== undefined) {
|
||||
defineAcpxResponseProperty(
|
||||
content,
|
||||
binding.name,
|
||||
binding.optionValues.get(selected) === "true",
|
||||
);
|
||||
}
|
||||
} else {
|
||||
const selected = (answer.selectedOptionIds ?? []).map((id) => binding.optionValues.get(id));
|
||||
if (binding.type === "array") {
|
||||
defineAcpxResponseProperty(content, binding.name, selected);
|
||||
} else if (selected[0] !== undefined) {
|
||||
defineAcpxResponseProperty(content, binding.name, selected[0]);
|
||||
}
|
||||
}
|
||||
}
|
||||
return { questionSet, nativeResponse: { action: "accept", content } };
|
||||
}
|
||||
|
||||
function fixtureAnswerHasValue(answer) {
|
||||
return (
|
||||
(typeof answer.text === "string" && answer.text.trim().length > 0)
|
||||
|| (typeof answer.customText === "string" && answer.customText.trim().length > 0)
|
||||
|| (Array.isArray(answer.selectedOptionIds) && answer.selectedOptionIds.length > 0)
|
||||
);
|
||||
}
|
||||
|
||||
function defineAcpxResponseProperty(content, name, value) {
|
||||
Object.defineProperty(content, name, {
|
||||
value,
|
||||
enumerable: true,
|
||||
configurable: true,
|
||||
writable: true,
|
||||
});
|
||||
}
|
||||
|
||||
function normalizedAcpxRequired(schema) {
|
||||
const propertyNames = new Set(
|
||||
isPlainRecord(schema.properties) ? Object.keys(schema.properties) : [],
|
||||
);
|
||||
return new Set(
|
||||
Array.isArray(schema.required)
|
||||
? schema.required
|
||||
.filter(
|
||||
(value) => typeof value === "string" && propertyNames.has(value),
|
||||
)
|
||||
: []
|
||||
);
|
||||
}
|
||||
|
||||
function projectAcpxProperty(name, property, index, required) {
|
||||
const id = stableAcpxFieldId(name, index);
|
||||
const header = optionalFixtureText(property.title) ?? name;
|
||||
const base = {
|
||||
id,
|
||||
header,
|
||||
prompt: optionalFixtureText(property.description) ?? header,
|
||||
required,
|
||||
};
|
||||
if (property.type === "string") {
|
||||
const options = acpxNativeOptions(property, "oneOf");
|
||||
if (options.length > 0) return projectedAcpxOptions(name, property.type, base, options, "single_select");
|
||||
const pattern = optionalFixtureText(property.pattern);
|
||||
return {
|
||||
name,
|
||||
type: property.type,
|
||||
optionValues: new Map(),
|
||||
question: {
|
||||
...base,
|
||||
answerMode: "text",
|
||||
textValidation: {
|
||||
inputType: "text",
|
||||
...(finiteNonNegativeFixtureInteger(property.minLength) !== undefined
|
||||
? { minLength: property.minLength }
|
||||
: {}),
|
||||
...(finiteNonNegativeFixtureInteger(property.maxLength) !== undefined
|
||||
? { maxLength: property.maxLength }
|
||||
: {}),
|
||||
...(pattern !== undefined ? { pattern } : {}),
|
||||
},
|
||||
},
|
||||
};
|
||||
}
|
||||
if (property.type === "number" || property.type === "integer") {
|
||||
return {
|
||||
name,
|
||||
type: property.type,
|
||||
optionValues: new Map(),
|
||||
question: {
|
||||
...base,
|
||||
answerMode: "text",
|
||||
textValidation: {
|
||||
inputType: property.type,
|
||||
...(Number.isFinite(property.minimum) ? { minimum: property.minimum } : {}),
|
||||
...(Number.isFinite(property.maximum) ? { maximum: property.maximum } : {}),
|
||||
},
|
||||
},
|
||||
};
|
||||
}
|
||||
if (property.type === "boolean") {
|
||||
return projectedAcpxOptions(name, property.type, base, [
|
||||
{ value: "true", label: "Yes" },
|
||||
{ value: "false", label: "No" },
|
||||
], "single_select");
|
||||
}
|
||||
return projectedAcpxOptions(
|
||||
name,
|
||||
property.type,
|
||||
base,
|
||||
acpxNativeOptions(property.items, "anyOf"),
|
||||
"multi_select",
|
||||
);
|
||||
}
|
||||
|
||||
function testFixturePatternWithDeadline(pattern, value, questionId) {
|
||||
const outcome = spawnSync(
|
||||
process.execPath,
|
||||
[
|
||||
"--input-type=module",
|
||||
"--eval",
|
||||
"const chunks=[];for await(const chunk of process.stdin)chunks.push(chunk);const {pattern,value}=JSON.parse(Buffer.concat(chunks).toString());process.stdout.write(new RegExp(pattern).test(value)?'1':'0');",
|
||||
],
|
||||
{
|
||||
input: JSON.stringify({ pattern, value }),
|
||||
encoding: "utf8",
|
||||
timeout: 1_000,
|
||||
maxBuffer: 1_024,
|
||||
windowsHide: true,
|
||||
},
|
||||
);
|
||||
if (outcome.error || outcome.signal || outcome.status !== 0) {
|
||||
throw contractError(
|
||||
"invalid_question_adapter_fixture",
|
||||
`text validation pattern for ${questionId} could not be evaluated safely`,
|
||||
);
|
||||
}
|
||||
return outcome.stdout === "1";
|
||||
}
|
||||
|
||||
function projectedAcpxOptions(name, type, base, nativeOptions, answerMode) {
|
||||
const optionValues = new Map();
|
||||
const options = nativeOptions.map((option, index) => {
|
||||
const id = `option-${index + 1}`;
|
||||
optionValues.set(id, option.value);
|
||||
return {
|
||||
id,
|
||||
label: option.label,
|
||||
...(option.description !== undefined ? { description: option.description } : {}),
|
||||
};
|
||||
});
|
||||
return { name, type, optionValues, question: { ...base, answerMode, options } };
|
||||
}
|
||||
|
||||
function acpxNativeOptions(property, preferredTitledKey) {
|
||||
const alternateTitledKey = preferredTitledKey === "anyOf" ? "oneOf" : "anyOf";
|
||||
const titled = property[preferredTitledKey]
|
||||
?? property[alternateTitledKey];
|
||||
if (Array.isArray(titled)) {
|
||||
return titled.map((entry) => ({
|
||||
value: entry.const,
|
||||
label: optionalFixtureText(entry.title) ?? entry.const,
|
||||
...(optionalFixtureText(entry.description) !== undefined
|
||||
? { description: entry.description }
|
||||
: {}),
|
||||
}));
|
||||
}
|
||||
return (Array.isArray(property.enum) ? property.enum : [])
|
||||
.map((value) => ({ value, label: value }));
|
||||
}
|
||||
|
||||
function stableAcpxFieldId(value, index) {
|
||||
const readable = value
|
||||
.trim()
|
||||
.replace(/[^A-Za-z0-9._:-]+/g, "-")
|
||||
.replace(/^-+|-+$/g, "")
|
||||
.slice(0, 96);
|
||||
return `field-${index + 1}-${readable || "value"}-${sha256(value).slice(0, 12)}`;
|
||||
}
|
||||
|
||||
function optionalFixtureText(value) {
|
||||
return typeof value === "string" && value.length > 0 ? value : undefined;
|
||||
}
|
||||
|
||||
function finiteNonNegativeFixtureInteger(value) {
|
||||
return Number.isSafeInteger(value) && value >= 0 ? value : undefined;
|
||||
}
|
||||
|
||||
function canonicalFixtureJson(value) {
|
||||
if (Array.isArray(value)) return `[${value.map(canonicalFixtureJson).join(",")}]`;
|
||||
if (isPlainRecord(value)) {
|
||||
return `{${Object.keys(value).sort().map((key) =>
|
||||
`${JSON.stringify(key)}:${canonicalFixtureJson(value[key])}`
|
||||
).join(",")}}`;
|
||||
}
|
||||
return JSON.stringify(value);
|
||||
}
|
||||
|
||||
function assertAcpxProperty(name, value) {
|
||||
if (!isPlainRecord(value)) {
|
||||
throw contractError("invalid_acpx_question_fixture", `native property ${name}`);
|
||||
}
|
||||
if (!["string", "number", "integer", "boolean", "array"].includes(value.type)) {
|
||||
throw contractError("invalid_acpx_question_fixture", `unsupported native property ${name}`);
|
||||
}
|
||||
if (value.type === "array") {
|
||||
if (!isPlainRecord(value.items) || acpxEnumValues(value.items, "anyOf").length === 0) {
|
||||
throw contractError("invalid_acpx_question_fixture", `native array property ${name} requires options`);
|
||||
}
|
||||
} else if (value.type === "string") {
|
||||
acpxEnumValues(value, "oneOf");
|
||||
}
|
||||
}
|
||||
|
||||
function assertAcpxPropertyValue(name, property, value) {
|
||||
const typeMatches = property.type === "string"
|
||||
? typeof value === "string"
|
||||
: property.type === "number"
|
||||
? typeof value === "number" && Number.isFinite(value)
|
||||
: property.type === "integer"
|
||||
? Number.isInteger(value)
|
||||
: property.type === "boolean"
|
||||
? typeof value === "boolean"
|
||||
: Array.isArray(value) && value.every((item) => typeof item === "string");
|
||||
if (!typeMatches) {
|
||||
throw contractError("invalid_acpx_question_fixture", `native response type for ${name}`);
|
||||
}
|
||||
const enumValues = property.type === "array"
|
||||
? acpxEnumValues(property.items, "anyOf")
|
||||
: property.type === "string"
|
||||
? acpxEnumValues(property, "oneOf")
|
||||
: [];
|
||||
const responseValues = Array.isArray(value) ? value : [value];
|
||||
if (enumValues.length > 0 && responseValues.some((item) => !enumValues.includes(item))) {
|
||||
throw contractError("invalid_acpx_question_fixture", `native response option for ${name}`);
|
||||
}
|
||||
}
|
||||
|
||||
function acpxEnumValues(property, preferredTitledKey) {
|
||||
if (!isPlainRecord(property)) return [];
|
||||
const alternateTitledKey = preferredTitledKey === "anyOf" ? "oneOf" : "anyOf";
|
||||
const titled = property[preferredTitledKey]
|
||||
?? property[alternateTitledKey];
|
||||
const values = Array.isArray(titled)
|
||||
? titled.map((entry) => isPlainRecord(entry) ? entry.const : undefined)
|
||||
: Array.isArray(property.enum)
|
||||
? property.enum
|
||||
: [];
|
||||
if (
|
||||
!Array.isArray(values)
|
||||
|| values.length > 128
|
||||
|| values.some((value) => typeof value !== "string" || value.length === 0)
|
||||
) {
|
||||
throw contractError("invalid_acpx_question_fixture", "native options must be non-empty bounded strings");
|
||||
}
|
||||
return values;
|
||||
}
|
||||
|
||||
function isPlainRecord(value) {
|
||||
return typeof value === "object" && value !== null && !Array.isArray(value);
|
||||
}
|
||||
|
||||
export function assertConformanceFixturePair(fixture, output) {
|
||||
if (fixture?.schemaVersion !== "paperclip.runner.conformance.fixture.v1") {
|
||||
throw contractError("unsupported_required_schema", `conformance fixture requires ${String(fixture?.schemaVersion)}`);
|
||||
|
|
|
|||
|
|
@ -0,0 +1,136 @@
|
|||
import { readFile } from "node:fs/promises";
|
||||
|
||||
import type { AcpElicitationRequest } from "acpx/runtime";
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import { normalizeAcpFormElicitation } from "../drivers/acpx/acp-question-adapter.js";
|
||||
import {
|
||||
createCodexQuestionResponseContext,
|
||||
normalizeCodexQuestionSet,
|
||||
runtimeRequestResponse,
|
||||
} from "../drivers/codex/codex-question-adapter.js";
|
||||
import {
|
||||
parsePaperclipQuestionResponse,
|
||||
parsePaperclipQuestionSet,
|
||||
type PaperclipQuestionSet,
|
||||
} from "./question-set.js";
|
||||
import type { HarnessRuntimeRequest } from "./harness-driver.js";
|
||||
|
||||
interface QuestionAdapterFixture {
|
||||
schema: "paperclip.question_adapter_fixture.v1";
|
||||
adapter: "codex" | "acpx";
|
||||
nativeRequest: Record<string, unknown>;
|
||||
canonicalQuestionSet: unknown;
|
||||
canonicalResponse: unknown;
|
||||
nativeResponse: unknown;
|
||||
}
|
||||
|
||||
async function fixture(
|
||||
adapter: QuestionAdapterFixture["adapter"],
|
||||
): Promise<QuestionAdapterFixture> {
|
||||
const source = await readFile(
|
||||
new URL(
|
||||
`../../protocol/fixtures/questions/${adapter}.json`,
|
||||
import.meta.url,
|
||||
),
|
||||
"utf8",
|
||||
);
|
||||
return JSON.parse(source) as QuestionAdapterFixture;
|
||||
}
|
||||
|
||||
describe("question adapter conformance fixtures", () => {
|
||||
it("normalizes equivalent Codex and ACPX forms to one canonical question set", async () => {
|
||||
const [codex, acpx] = await Promise.all([
|
||||
fixture("codex"),
|
||||
fixture("acpx"),
|
||||
]);
|
||||
const codexQuestionSet = normalizeCodexQuestionSet(
|
||||
String(codex.nativeRequest.method),
|
||||
codex.nativeRequest.params as Record<string, unknown>,
|
||||
createCodexQuestionResponseContext(),
|
||||
);
|
||||
const acpxQuestionSet = normalizeAcpFormElicitation(
|
||||
acpx.nativeRequest.params as AcpElicitationRequest,
|
||||
)?.questionSet;
|
||||
const codexExpected = parsePaperclipQuestionSet(codex.canonicalQuestionSet);
|
||||
const acpxExpected = parsePaperclipQuestionSet(acpx.canonicalQuestionSet);
|
||||
|
||||
expect(codexQuestionSet).toEqual(codexExpected);
|
||||
expect(acpxQuestionSet).toEqual(acpxExpected);
|
||||
expect(questionPresentation(acpxExpected)).toEqual(
|
||||
questionPresentation(codexExpected),
|
||||
);
|
||||
expect(
|
||||
parsePaperclipQuestionResponse(codexExpected, codex.canonicalResponse),
|
||||
).toEqual(codex.canonicalResponse);
|
||||
expect(
|
||||
parsePaperclipQuestionResponse(acpxExpected, acpx.canonicalResponse),
|
||||
).toEqual(acpx.canonicalResponse);
|
||||
});
|
||||
|
||||
it("converts one canonical response back to each provider shape", async () => {
|
||||
const [codex, acpx] = await Promise.all([
|
||||
fixture("codex"),
|
||||
fixture("acpx"),
|
||||
]);
|
||||
const responseContext = createCodexQuestionResponseContext();
|
||||
const codexQuestionSet = normalizeCodexQuestionSet(
|
||||
String(codex.nativeRequest.method),
|
||||
codex.nativeRequest.params as Record<string, unknown>,
|
||||
responseContext,
|
||||
);
|
||||
if (codexQuestionSet === null) {
|
||||
throw new Error("Codex conformance fixture did not contain a question set");
|
||||
}
|
||||
const codexRequest: HarnessRuntimeRequest = {
|
||||
requestId: "fixture-request",
|
||||
requestKind: "user_input",
|
||||
method: String(codex.nativeRequest.method),
|
||||
turnId: "fixture-turn",
|
||||
itemId: "fixture-item",
|
||||
status: "pending",
|
||||
prompt: "Deployment input",
|
||||
details: {},
|
||||
input: codexQuestionSet,
|
||||
origin: {
|
||||
adapter: "codex",
|
||||
method: String(codex.nativeRequest.method),
|
||||
},
|
||||
};
|
||||
const codexResponse = parsePaperclipQuestionResponse(
|
||||
codexQuestionSet,
|
||||
codex.canonicalResponse,
|
||||
);
|
||||
const normalizedAcpx = normalizeAcpFormElicitation(
|
||||
acpx.nativeRequest.params as AcpElicitationRequest,
|
||||
);
|
||||
const acpxResponse = parsePaperclipQuestionResponse(
|
||||
normalizedAcpx!.questionSet,
|
||||
acpx.canonicalResponse,
|
||||
);
|
||||
|
||||
expect(
|
||||
runtimeRequestResponse(codexRequest, {
|
||||
action: "submit",
|
||||
response: codexResponse,
|
||||
}, responseContext),
|
||||
).toEqual(codex.nativeResponse);
|
||||
expect(normalizedAcpx?.accept(acpxResponse)).toEqual(acpx.nativeResponse);
|
||||
});
|
||||
});
|
||||
|
||||
function questionPresentation(questionSet: PaperclipQuestionSet): unknown {
|
||||
return {
|
||||
...questionSet,
|
||||
questions: questionSet.questions.map(
|
||||
({ id: _questionId, options, ...question }) => ({
|
||||
...question,
|
||||
...(options
|
||||
? {
|
||||
options: options.map(({ id: _optionId, ...option }) => option),
|
||||
}
|
||||
: {}),
|
||||
}),
|
||||
),
|
||||
};
|
||||
}
|
||||
|
|
@ -1,4 +1,5 @@
|
|||
import assert from "node:assert/strict";
|
||||
import { createHash } from "node:crypto";
|
||||
import { readFile } from "node:fs/promises";
|
||||
import { dirname, resolve } from "node:path";
|
||||
import test from "node:test";
|
||||
|
|
@ -6,8 +7,10 @@ import { fileURLToPath } from "node:url";
|
|||
|
||||
import { buildProtocolManifest } from "../scripts/generate-protocol-manifest.mjs";
|
||||
import {
|
||||
assertAcpxQuestionFixture,
|
||||
assertCodexQuestionFixture,
|
||||
assertConformanceFixturePair,
|
||||
assertQuestionAdapterFixture,
|
||||
assertReplayFixtureCompatibility,
|
||||
assertSchemaInstance,
|
||||
compileProtocolValidators,
|
||||
|
|
@ -112,6 +115,495 @@ test("the Codex question fixture uses stable provider-neutral IDs", async () =>
|
|||
assert.deepEqual(Object.keys(value.canonicalResponse.answers), ["environment"]);
|
||||
});
|
||||
|
||||
test("the ACPX question fixture enforces its provider-native contract", async () => {
|
||||
const canonical = await fixture("questions/acpx.json");
|
||||
assert.doesNotThrow(() => assertAcpxQuestionFixture(canonical));
|
||||
|
||||
const implicitObjectSchema = structuredClone(canonical);
|
||||
delete implicitObjectSchema.nativeRequest.params.requestedSchema.type;
|
||||
assert.doesNotThrow(() => assertAcpxQuestionFixture(implicitObjectSchema));
|
||||
|
||||
const unionObjectSchema = structuredClone(canonical);
|
||||
unionObjectSchema.nativeRequest.params.requestedSchema.type = ["object", "null"];
|
||||
assert.doesNotThrow(() => assertAcpxQuestionFixture(unionObjectSchema));
|
||||
|
||||
const malformedRequest = structuredClone(canonical);
|
||||
malformedRequest.nativeRequest.params.requestedSchema.properties.environment.type = "object";
|
||||
assert.throws(
|
||||
() => assertAcpxQuestionFixture(malformedRequest),
|
||||
/unsupported native property environment/,
|
||||
);
|
||||
|
||||
const malformedResponse = structuredClone(canonical);
|
||||
malformedResponse.nativeResponse.content.environment = "unknown";
|
||||
assert.throws(
|
||||
() => assertAcpxQuestionFixture(malformedResponse),
|
||||
/native response option for environment/,
|
||||
);
|
||||
|
||||
const patternBearingQuestion = structuredClone(canonical);
|
||||
const patternBearingProperty = patternBearingQuestion.nativeRequest.params
|
||||
.requestedSchema.properties.environment;
|
||||
delete patternBearingProperty.oneOf;
|
||||
patternBearingProperty.pattern = "^staging$";
|
||||
patternBearingQuestion.canonicalQuestionSet.questions[0] = {
|
||||
id: "field-1-environment-ba5285161ba6",
|
||||
header: "Environment",
|
||||
prompt: "Where should we deploy?",
|
||||
required: true,
|
||||
answerMode: "text",
|
||||
textValidation: { inputType: "text", pattern: "^staging$" },
|
||||
};
|
||||
patternBearingQuestion.canonicalResponse.answers = {
|
||||
"field-1-environment-ba5285161ba6": { text: "staging" },
|
||||
};
|
||||
assert.throws(
|
||||
() => assertAcpxQuestionFixture(patternBearingQuestion),
|
||||
/native property environment uses an unsupported pattern/,
|
||||
);
|
||||
|
||||
const optionPattern = structuredClone(canonical);
|
||||
optionPattern.nativeRequest.params.requestedSchema.properties.environment
|
||||
.pattern = "^staging$";
|
||||
assert.doesNotThrow(() => assertAcpxQuestionFixture(optionPattern));
|
||||
|
||||
const emptyOptionalAnswer = structuredClone(canonical);
|
||||
emptyOptionalAnswer.nativeRequest.params.requestedSchema.required = [];
|
||||
emptyOptionalAnswer.nativeResponse.content = {};
|
||||
emptyOptionalAnswer.canonicalQuestionSet.questions[0] = {
|
||||
...emptyOptionalAnswer.canonicalQuestionSet.questions[0],
|
||||
required: false,
|
||||
};
|
||||
emptyOptionalAnswer.canonicalResponse.answers = {
|
||||
[emptyOptionalAnswer.canonicalQuestionSet.questions[0].id]: {},
|
||||
};
|
||||
assert.doesNotThrow(() => assertAcpxQuestionFixture(emptyOptionalAnswer));
|
||||
|
||||
const emptyOption = structuredClone(canonical);
|
||||
emptyOption.nativeRequest.params.requestedSchema.properties.environment
|
||||
.oneOf[0].const = "";
|
||||
assert.throws(
|
||||
() => assertAcpxQuestionFixture(emptyOption),
|
||||
/native options must be non-empty bounded strings/,
|
||||
);
|
||||
|
||||
const missingResponse = structuredClone(canonical);
|
||||
missingResponse.nativeResponse.content = {};
|
||||
assert.throws(
|
||||
() => assertAcpxQuestionFixture(missingResponse),
|
||||
/native response omits required property environment/,
|
||||
);
|
||||
|
||||
const divergentQuestion = structuredClone(canonical);
|
||||
divergentQuestion.nativeRequest.params.requestedSchema.required = [];
|
||||
assert.throws(
|
||||
() => assertAcpxQuestionFixture(divergentQuestion),
|
||||
/native and canonical question sets differ/,
|
||||
);
|
||||
|
||||
const normalizedRequired = structuredClone(canonical);
|
||||
normalizedRequired.nativeRequest.params.requestedSchema.required = [
|
||||
"environment",
|
||||
"environment",
|
||||
42,
|
||||
"not-a-property",
|
||||
];
|
||||
assert.doesNotThrow(() => assertAcpxQuestionFixture(normalizedRequired));
|
||||
|
||||
const manyUnknownRequired = structuredClone(canonical);
|
||||
manyUnknownRequired.nativeRequest.params.requestedSchema.required = [
|
||||
...Array.from({ length: 80 }, (_, index) => `unknown-${index}`),
|
||||
"environment",
|
||||
];
|
||||
assert.doesNotThrow(() => assertAcpxQuestionFixture(manyUnknownRequired));
|
||||
|
||||
const divergentResponse = structuredClone(canonical);
|
||||
divergentResponse.nativeResponse.content.environment = "production";
|
||||
assert.throws(
|
||||
() => assertAcpxQuestionFixture(divergentResponse),
|
||||
/canonical and native responses differ/,
|
||||
);
|
||||
|
||||
const arrayWithBothOptionForms = structuredClone(canonical);
|
||||
const environment = arrayWithBothOptionForms.nativeRequest.params
|
||||
.requestedSchema.properties.environment;
|
||||
environment.type = "array";
|
||||
environment.items = {
|
||||
anyOf: environment.oneOf,
|
||||
oneOf: [{ const: "wrong", title: "Wrong precedence" }],
|
||||
};
|
||||
delete environment.oneOf;
|
||||
arrayWithBothOptionForms.canonicalQuestionSet.questions[0].answerMode = "multi_select";
|
||||
arrayWithBothOptionForms.nativeResponse.content.environment = ["staging"];
|
||||
assert.doesNotThrow(() => assertAcpxQuestionFixture(arrayWithBothOptionForms));
|
||||
|
||||
const malformedPreferredUnion = structuredClone(arrayWithBothOptionForms);
|
||||
malformedPreferredUnion.nativeRequest.params.requestedSchema.properties
|
||||
.environment.items.anyOf = { const: "staging" };
|
||||
assert.throws(
|
||||
() => assertAcpxQuestionFixture(malformedPreferredUnion),
|
||||
/native array property environment requires options/,
|
||||
);
|
||||
|
||||
const enumFallback = structuredClone(malformedPreferredUnion);
|
||||
enumFallback.nativeRequest.params.requestedSchema.properties.environment.items = {
|
||||
anyOf: { const: "ignored-malformed-preferred-union" },
|
||||
oneOf: [{ const: "wrong", title: "Wrong alternate union" }],
|
||||
enum: ["staging", "production"],
|
||||
};
|
||||
enumFallback.canonicalQuestionSet.questions[0].options = [
|
||||
{ id: "option-1", label: "staging" },
|
||||
{ id: "option-2", label: "production" },
|
||||
];
|
||||
assert.doesNotThrow(() => assertAcpxQuestionFixture(enumFallback));
|
||||
|
||||
const specialPropertyName = "__proto__";
|
||||
const specialProperty = structuredClone(
|
||||
canonical.nativeRequest.params.requestedSchema.properties.environment,
|
||||
);
|
||||
const specialPropertyFixture = structuredClone(canonical);
|
||||
const specialProperties = specialPropertyFixture.nativeRequest.params
|
||||
.requestedSchema.properties;
|
||||
delete specialProperties.environment;
|
||||
Object.defineProperty(specialProperties, specialPropertyName, {
|
||||
value: specialProperty,
|
||||
enumerable: true,
|
||||
configurable: true,
|
||||
writable: true,
|
||||
});
|
||||
specialPropertyFixture.nativeRequest.params.requestedSchema.required = [
|
||||
specialPropertyName,
|
||||
];
|
||||
const specialQuestionId = `field-1-__proto__-${createHash("sha256")
|
||||
.update(specialPropertyName)
|
||||
.digest("hex")
|
||||
.slice(0, 12)}`;
|
||||
specialPropertyFixture.canonicalQuestionSet.questions[0].id = specialQuestionId;
|
||||
specialPropertyFixture.canonicalResponse.answers = {
|
||||
[specialQuestionId]: { selectedOptionIds: ["option-1"] },
|
||||
};
|
||||
specialPropertyFixture.nativeResponse.content = {};
|
||||
Object.defineProperty(
|
||||
specialPropertyFixture.nativeResponse.content,
|
||||
specialPropertyName,
|
||||
{
|
||||
value: "staging",
|
||||
enumerable: true,
|
||||
configurable: true,
|
||||
writable: true,
|
||||
},
|
||||
);
|
||||
assert.doesNotThrow(() => assertAcpxQuestionFixture(specialPropertyFixture));
|
||||
});
|
||||
|
||||
test("the ACPX question fixture ignores scalar enums the adapter does not project", async () => {
|
||||
const base = await fixture("questions/acpx.json");
|
||||
const id = "field-1-environment-ba5285161ba6";
|
||||
const cases = [
|
||||
{
|
||||
type: "boolean",
|
||||
property: { type: "boolean", title: "Confirm", enum: [true] },
|
||||
question: {
|
||||
id,
|
||||
header: "Confirm",
|
||||
prompt: "Confirm",
|
||||
required: true,
|
||||
answerMode: "single_select",
|
||||
options: [
|
||||
{ id: "option-1", label: "Yes" },
|
||||
{ id: "option-2", label: "No" },
|
||||
],
|
||||
},
|
||||
canonicalAnswer: { selectedOptionIds: ["option-2"] },
|
||||
nativeAnswer: false,
|
||||
},
|
||||
{
|
||||
type: "number",
|
||||
property: { type: "number", title: "Threshold", enum: [1.5] },
|
||||
question: {
|
||||
id,
|
||||
header: "Threshold",
|
||||
prompt: "Threshold",
|
||||
required: true,
|
||||
answerMode: "text",
|
||||
textValidation: { inputType: "number" },
|
||||
},
|
||||
canonicalAnswer: { text: "2.5" },
|
||||
nativeAnswer: 2.5,
|
||||
},
|
||||
{
|
||||
type: "integer",
|
||||
property: { type: "integer", title: "Replicas", enum: [1] },
|
||||
question: {
|
||||
id,
|
||||
header: "Replicas",
|
||||
prompt: "Replicas",
|
||||
required: true,
|
||||
answerMode: "text",
|
||||
textValidation: { inputType: "integer" },
|
||||
},
|
||||
canonicalAnswer: { text: "2" },
|
||||
nativeAnswer: 2,
|
||||
},
|
||||
];
|
||||
|
||||
for (const testCase of cases) {
|
||||
const value = structuredClone(base);
|
||||
value.nativeRequest.params.requestedSchema.properties.environment = testCase.property;
|
||||
value.canonicalQuestionSet.questions[0] = testCase.question;
|
||||
value.canonicalResponse.answers[id] = testCase.canonicalAnswer;
|
||||
value.nativeResponse.content.environment = testCase.nativeAnswer;
|
||||
assert.doesNotThrow(
|
||||
() => assertAcpxQuestionFixture(value),
|
||||
`${testCase.type} enum must not constrain the adapter's projected scalar answer`,
|
||||
);
|
||||
}
|
||||
|
||||
const invalidNativeType = structuredClone(base);
|
||||
invalidNativeType.nativeRequest.params.requestedSchema.properties.environment =
|
||||
cases[0].property;
|
||||
invalidNativeType.canonicalQuestionSet.questions[0] = cases[0].question;
|
||||
invalidNativeType.canonicalResponse.answers[id] = cases[0].canonicalAnswer;
|
||||
invalidNativeType.nativeResponse.content.environment = "false";
|
||||
assert.throws(
|
||||
() => assertAcpxQuestionFixture(invalidNativeType),
|
||||
/native response type for environment/,
|
||||
);
|
||||
});
|
||||
|
||||
test("every question adapter fixture satisfies its declared schema", async () => {
|
||||
const schemas = await loadSchemaCatalog(resolve(protocolRoot, "schemas"));
|
||||
const validators = compileProtocolValidators(schemas);
|
||||
for (const adapter of ["codex", "acpx"]) {
|
||||
const value = await fixture(`questions/${adapter}.json`);
|
||||
assert.doesNotThrow(() =>
|
||||
assertSchemaInstance(
|
||||
validators.questionAdapterFixture,
|
||||
value,
|
||||
`${adapter}-question-fixture`,
|
||||
),
|
||||
);
|
||||
assert.doesNotThrow(() => assertQuestionAdapterFixture(value));
|
||||
}
|
||||
|
||||
const malformed = structuredClone(await fixture("questions/acpx.json"));
|
||||
delete malformed.canonicalQuestionSet.schema;
|
||||
assert.throws(
|
||||
() =>
|
||||
assertSchemaInstance(
|
||||
validators.questionAdapterFixture,
|
||||
malformed,
|
||||
"malformed-acpx-question-fixture",
|
||||
),
|
||||
/schema_validation_failed/,
|
||||
);
|
||||
|
||||
const unknownQuestion = structuredClone(await fixture("questions/acpx.json"));
|
||||
unknownQuestion.canonicalResponse.answers = {
|
||||
"unknown-question": { selectedOptionIds: ["option-1"] },
|
||||
};
|
||||
assert.throws(
|
||||
() => assertQuestionAdapterFixture(unknownQuestion),
|
||||
/answer has unknown question ID unknown-question/,
|
||||
);
|
||||
|
||||
const unknownOption = structuredClone(await fixture("questions/acpx.json"));
|
||||
const [questionId] = Object.keys(unknownOption.canonicalResponse.answers);
|
||||
unknownOption.canonicalResponse.answers[questionId].selectedOptionIds = [
|
||||
"unknown-option",
|
||||
];
|
||||
assert.throws(
|
||||
() => assertQuestionAdapterFixture(unknownOption),
|
||||
/answer has unknown option ID unknown-option/,
|
||||
);
|
||||
|
||||
const canonical = await fixture("questions/acpx.json");
|
||||
const [requiredQuestion] = canonical.canonicalQuestionSet.questions;
|
||||
const requiredQuestionId = requiredQuestion.id;
|
||||
const malformedAnswers = [
|
||||
{
|
||||
label: "missing required answer",
|
||||
answer: undefined,
|
||||
pattern: /required question .* is missing/,
|
||||
},
|
||||
{
|
||||
label: "empty required answer",
|
||||
answer: {},
|
||||
pattern: /required question .* is empty/,
|
||||
},
|
||||
{
|
||||
label: "multiple single-select values",
|
||||
answer: { selectedOptionIds: requiredQuestion.options.map((option) => option.id) },
|
||||
pattern: /single-select answer .* chooses more than one option/,
|
||||
},
|
||||
{
|
||||
label: "text on a select question",
|
||||
answer: { text: "staging" },
|
||||
pattern: /select answer .* carries text/,
|
||||
},
|
||||
{
|
||||
label: "disabled custom answer",
|
||||
answer: { customText: "canary" },
|
||||
pattern: /custom answer .* is not enabled/,
|
||||
},
|
||||
];
|
||||
for (const malformedAnswer of malformedAnswers) {
|
||||
const malformedResponse = structuredClone(canonical);
|
||||
malformedResponse.canonicalResponse.answers = malformedAnswer.answer === undefined
|
||||
? {}
|
||||
: { [requiredQuestionId]: malformedAnswer.answer };
|
||||
assert.throws(
|
||||
() => assertQuestionAdapterFixture(malformedResponse),
|
||||
malformedAnswer.pattern,
|
||||
malformedAnswer.label,
|
||||
);
|
||||
}
|
||||
|
||||
const emptyOptionalAnswer = structuredClone(canonical);
|
||||
emptyOptionalAnswer.canonicalQuestionSet.questions[0] = {
|
||||
...requiredQuestion,
|
||||
required: false,
|
||||
};
|
||||
emptyOptionalAnswer.canonicalResponse.answers = {
|
||||
[requiredQuestionId]: {},
|
||||
};
|
||||
assert.doesNotThrow(() => assertQuestionAdapterFixture(emptyOptionalAnswer));
|
||||
|
||||
const textModeMismatch = structuredClone(canonical);
|
||||
textModeMismatch.canonicalQuestionSet.questions[0] = {
|
||||
...requiredQuestion,
|
||||
answerMode: "text",
|
||||
options: [],
|
||||
};
|
||||
textModeMismatch.canonicalResponse.answers[requiredQuestionId] = {
|
||||
customText: "select-only custom value",
|
||||
};
|
||||
assert.throws(
|
||||
() => assertQuestionAdapterFixture(textModeMismatch),
|
||||
/text answer .* carries select-only fields/,
|
||||
);
|
||||
|
||||
const invalidTextAnswers = [
|
||||
{
|
||||
label: "minimum text length",
|
||||
validation: { minLength: 4 },
|
||||
text: "abc",
|
||||
pattern: /must contain at least 4 characters/,
|
||||
},
|
||||
{
|
||||
label: "maximum text length",
|
||||
validation: { maxLength: 2 },
|
||||
text: "abc",
|
||||
pattern: /must contain at most 2 characters/,
|
||||
},
|
||||
{
|
||||
label: "text pattern",
|
||||
validation: { pattern: "^z+$" },
|
||||
text: "abc",
|
||||
pattern: /does not match the required format/,
|
||||
},
|
||||
{
|
||||
label: "numeric input",
|
||||
validation: { inputType: "number" },
|
||||
text: "not-a-number",
|
||||
pattern: /must be a valid number/,
|
||||
},
|
||||
{
|
||||
label: "integer input",
|
||||
validation: { inputType: "integer" },
|
||||
text: "1.5",
|
||||
pattern: /must be a valid integer/,
|
||||
},
|
||||
{
|
||||
label: "numeric minimum",
|
||||
validation: { inputType: "number", minimum: 2 },
|
||||
text: "1",
|
||||
pattern: /must be at least 2/,
|
||||
},
|
||||
{
|
||||
label: "numeric maximum",
|
||||
validation: { inputType: "number", maximum: 2 },
|
||||
text: "3",
|
||||
pattern: /must be at most 2/,
|
||||
},
|
||||
];
|
||||
for (const invalid of invalidTextAnswers) {
|
||||
const malformedResponse = structuredClone(canonical);
|
||||
malformedResponse.canonicalQuestionSet.questions[0] = {
|
||||
...requiredQuestion,
|
||||
answerMode: "text",
|
||||
options: [],
|
||||
textValidation: invalid.validation,
|
||||
};
|
||||
malformedResponse.canonicalResponse.answers[requiredQuestionId] = {
|
||||
text: invalid.text,
|
||||
};
|
||||
assert.throws(
|
||||
() => assertQuestionAdapterFixture(malformedResponse),
|
||||
invalid.pattern,
|
||||
invalid.label,
|
||||
);
|
||||
}
|
||||
|
||||
const invalidCustomText = structuredClone(canonical);
|
||||
invalidCustomText.canonicalQuestionSet.questions[0] = {
|
||||
...requiredQuestion,
|
||||
customAnswer: { enabled: true },
|
||||
textValidation: { minLength: 2, inputType: "text" },
|
||||
};
|
||||
invalidCustomText.canonicalResponse.answers[requiredQuestionId] = {
|
||||
customText: "x",
|
||||
};
|
||||
assert.throws(
|
||||
() => assertQuestionAdapterFixture(invalidCustomText),
|
||||
/must contain at least 2 characters/,
|
||||
"select custom text validation",
|
||||
);
|
||||
|
||||
for (const invalid of [
|
||||
{
|
||||
label: "contradictory optional text lengths",
|
||||
validation: { minLength: 3, maxLength: 2 },
|
||||
pattern: /minLength greater than maxLength/,
|
||||
},
|
||||
{
|
||||
label: "contradictory optional numeric bounds",
|
||||
validation: { inputType: "number", minimum: 3, maximum: 2 },
|
||||
pattern: /minimum greater than maximum/,
|
||||
},
|
||||
]) {
|
||||
const malformedQuestion = structuredClone(canonical);
|
||||
malformedQuestion.canonicalQuestionSet.questions[0] = {
|
||||
...requiredQuestion,
|
||||
required: false,
|
||||
answerMode: "text",
|
||||
options: [],
|
||||
textValidation: invalid.validation,
|
||||
};
|
||||
malformedQuestion.canonicalResponse.answers = {};
|
||||
assert.throws(
|
||||
() => assertQuestionAdapterFixture(malformedQuestion),
|
||||
invalid.pattern,
|
||||
invalid.label,
|
||||
);
|
||||
}
|
||||
|
||||
const pathologicalPattern = structuredClone(canonical);
|
||||
pathologicalPattern.canonicalQuestionSet.questions[0] = {
|
||||
...requiredQuestion,
|
||||
answerMode: "text",
|
||||
options: [],
|
||||
textValidation: { pattern: "(a+)+$" },
|
||||
};
|
||||
pathologicalPattern.canonicalResponse.answers[requiredQuestionId] = {
|
||||
text: `${"a".repeat(100_000)}!`,
|
||||
};
|
||||
assert.throws(
|
||||
() => assertQuestionAdapterFixture(pathologicalPattern),
|
||||
/could not be evaluated safely/,
|
||||
);
|
||||
});
|
||||
|
||||
test("the cross-language conformance input and output have one stable identity", async () => {
|
||||
const input = await fixture("conformance-minimal-run.json");
|
||||
const output = await fixture("conformance-expected-output.json");
|
||||
|
|
|
|||
Loading…
Reference in New Issue