test(runner): spell out terminal result fixture
This commit is contained in:
Dotta
parent
60c0c57638
commit
6d199c4c61
|
|
@ -381,7 +381,7 @@ export const runnerTasks: readonly RunnerTaskFixture[] = [
|
|||
"Complete this task in a single run.",
|
||||
`The exact marker also appears unescaped in the task title: PAPERCLIP_E2E_OK_${nonce}`,
|
||||
`Your final visible task-thread response must be exactly this marker: PAPERCLIP_E2E_OK_${nonce}`,
|
||||
`In a native runner, first call paperclip_finish with that marker as its summary and an objective-satisfied claim for the supplied contract revision. After it succeeds, emit that exact marker once as the complete user-facing final response and do not call another tool.`,
|
||||
`In a native runner, call paperclip_finish exactly once with {reportedWorkDisposition:"done",summary:"PAPERCLIP_E2E_OK_${nonce}",completionClaim:{contractRevision:"1",objectiveSatisfied:true,criteria:[{criterionId:"objective",status:"satisfied",evidenceRefs:[]}],remainingWork:[]},evidence:[],verification:[]}. After it succeeds, emit that exact marker once as the complete user-facing final response and do not call another tool.`,
|
||||
`In a legacy runner, make exactly one public-API write containing the marker: PATCH /api/issues/$PAPERCLIP_TASK_ID with {"status":"done","comment":"PAPERCLIP_E2E_OK_${nonce}"}. Do not POST to /comments, and do not include the marker in any other write.`,
|
||||
"The visible task-thread response is asserted; hidden reasoning or provider terminal output alone does not count.",
|
||||
"Use underscore characters exactly as shown and do not insert backslashes.",
|
||||
|
|
@ -448,7 +448,7 @@ export const runnerTasks: readonly RunnerTaskFixture[] = [
|
|||
"Only after the revised plan is accepted, implement it by posting one final visible task-thread response containing exactly " +
|
||||
`PAPERCLIP_E2E_PLAN_DONE_${nonce}` +
|
||||
" and mark the task Done.",
|
||||
`For a native runner, remain in the requested planning collaboration mode. Call write_document for key \`plan\`, then call request_human_input exactly once with interactionKind \`confirmation\`, targetRevisionId set to the returned latest Plan revision, and continuationPolicy \`wake_assignee\`. Do not call paperclip_finish while waiting for either Plan confirmation. After the revised Plan is accepted, first call paperclip_finish with PAPERCLIP_E2E_PLAN_DONE_${nonce} as its complete summary, then emit exactly that marker as the complete final response and do not call another tool.`,
|
||||
`For a native runner, remain in the requested planning collaboration mode. Call write_document for key \`plan\`, then call request_human_input exactly once with interactionKind \`confirmation\`, targetRevisionId set to the returned latest Plan revision, and continuationPolicy \`wake_assignee\`. Do not call paperclip_finish while waiting for either Plan confirmation. After the revised Plan is accepted, call paperclip_finish exactly once with {reportedWorkDisposition:"done",summary:"PAPERCLIP_E2E_PLAN_DONE_${nonce}",completionClaim:{contractRevision:"1",objectiveSatisfied:true,criteria:[{criterionId:"objective",status:"satisfied",evidenceRefs:[]}],remainingWork:[]},evidence:[],verification:[]}, then emit exactly that marker as the complete final response and do not call another tool.`,
|
||||
`For a legacy runner, use the public Paperclip API. The first PUT of the \`plan\` issue document creates it. For every later PUT, first GET the current document and set \`baseRevisionId\` to its \`latestRevisionId\`; a 409 means you must GET again and retry with the new latest revision. Create a \`request_confirmation\` targeting the successful PUT response's \`latestRevisionId\` with \`continuationPolicy: wake_assignee\`, and move the issue to \`in_review\` while waiting. After the revised Plan is accepted, write PAPERCLIP_E2E_PLAN_DONE_${nonce} exactly once through one atomic issue PATCH with status \`done\` and that exact comment; do not POST a separate comment or perform a second write.`,
|
||||
"Do not create files, child tasks, or unrelated work, and do not expose credentials.",
|
||||
].join("\n"),
|
||||
|
|
@ -499,7 +499,7 @@ export const runnerTasks: readonly RunnerTaskFixture[] = [
|
|||
`Your final visible task-thread answer must be exactly this complete marker, including its final suffix: E2E_ASK_12_${nonce}.`,
|
||||
"Do not create or modify files, do not create a plan or additional work, and do not expose credentials.",
|
||||
"After posting the direct answer, mark this task Done.",
|
||||
`In a native runner, first call paperclip_finish with E2E_ASK_12_${nonce} as the summary. After it succeeds, emit exactly E2E_ASK_12_${nonce} as the complete final response and do not call another tool.`,
|
||||
`In a native runner, call paperclip_finish exactly once with {reportedWorkDisposition:"done",summary:"E2E_ASK_12_${nonce}",completionClaim:{contractRevision:"1",objectiveSatisfied:true,criteria:[{criterionId:"objective",status:"satisfied",evidenceRefs:[]}],remainingWork:[]},evidence:[],verification:[]}. After it succeeds, emit exactly E2E_ASK_12_${nonce} as the complete final response and do not call another tool.`,
|
||||
`In a legacy runner, post exactly E2E_ASK_12_${nonce} as the task comment body and mark the task Done through the public API.`,
|
||||
].join("\n"),
|
||||
buildMatchers(nonce, execution) {
|
||||
|
|
@ -577,7 +577,7 @@ const structuredQuestionResumeTask = {
|
|||
`In a native runner, call request_human_input exactly once with idempotencyKey \`question-${nonce}\`, interactionKind \`questions\`, title \`Verification word\`, prompt \`Choose the verification word\`, continuationPolicy \`wake_assignee\`, and payload {version:1,questions:[{id:\`verification-word\`,prompt:\`Choose the verification word.\`,selectionMode:\`single\`,required:true,options:[{id:\`cobalt\`,label:\`Cobalt\`},{id:\`amber\`,label:\`Amber\`}]}]}.`,
|
||||
"In a legacy runner, POST /api/issues/$PAPERCLIP_TASK_ID/interactions with {kind:`ask_user_questions`,continuationPolicy:`wake_assignee`,payload:{version:1,questions:[{id:`verification-word`,prompt:`Choose the verification word.`,selectionMode:`single`,required:true,options:[{id:`cobalt`,label:`Cobalt`},{id:`amber`,label:`Amber`}]}]}} using Authorization and X-Paperclip-Run-Id, then move the issue to `in_review`.",
|
||||
`After the answer arrives, if it is Cobalt, publish exactly PAPERCLIP_E2E_QUESTION_DONE_${nonce} once as the complete visible response and mark the task Done.`,
|
||||
`In a native runner, first call paperclip_finish with PAPERCLIP_E2E_QUESTION_DONE_${nonce} as its summary, then emit that exact final response and do not call another tool.`,
|
||||
`In a native runner, after the answer arrives call paperclip_finish exactly once with {reportedWorkDisposition:"done",summary:"PAPERCLIP_E2E_QUESTION_DONE_${nonce}",completionClaim:{contractRevision:"1",objectiveSatisfied:true,criteria:[{criterionId:"objective",status:"satisfied",evidenceRefs:[]}],remainingWork:[]},evidence:[],verification:[]}, then emit that exact final response and do not call another tool.`,
|
||||
`In a legacy runner, post exactly PAPERCLIP_E2E_QUESTION_DONE_${nonce} as the task comment body and mark the task Done through the public API.`,
|
||||
"Do not create files, plans, child tasks, or unrelated work, and do not expose credentials.",
|
||||
].join("\n"),
|
||||
|
|
@ -636,7 +636,7 @@ export const openRouterBreadthTasks: readonly RunnerTaskFixture[] = [
|
|||
[
|
||||
"Complete this deterministic hello task in one turn.",
|
||||
`Return ${breadthMarker("H", nonce)} as the complete visible response.`,
|
||||
`Call paperclip_finish with ${breadthMarker("H", nonce)} as its summary and mark the task Done.`,
|
||||
`Call paperclip_finish exactly once with {reportedWorkDisposition:"done",summary:"${breadthMarker("H", nonce)}",completionClaim:{contractRevision:"1",objectiveSatisfied:true,criteria:[{criterionId:"objective",status:"satisfied",evidenceRefs:[]}],remainingWork:[]},evidence:[],verification:[]} and mark the task Done.`,
|
||||
"Do not create files, plans, interactions, or additional work.",
|
||||
].join("\n"),
|
||||
buildMatchers: (nonce, execution) =>
|
||||
|
|
@ -662,7 +662,7 @@ export const openRouterBreadthTasks: readonly RunnerTaskFixture[] = [
|
|||
"Ask the user one structured question before completing this task.",
|
||||
`Call request_human_input exactly once with idempotencyKey \`question-${nonce}\`, interactionKind \`questions\`, title \`Verification word\`, prompt \`Choose the verification word\`, continuationPolicy \`wake_assignee\`, and payload {version:1,questions:[{id:\`verification-word\`,prompt:\`Choose the verification word.\`,selectionMode:\`single\`,required:true,options:[{id:\`cobalt\`,label:\`Cobalt\`},{id:\`amber\`,label:\`Amber\`}]}]}.`,
|
||||
"Do not call paperclip_finish while the question is pending.",
|
||||
`After the answer arrives, if it is Cobalt, return ${breadthMarker("Q_C", nonce)} visibly and call paperclip_finish with that marker as the summary.`,
|
||||
`After the answer arrives, if it is Cobalt, return ${breadthMarker("Q_C", nonce)} visibly and call paperclip_finish exactly once with {reportedWorkDisposition:"done",summary:"${breadthMarker("Q_C", nonce)}",completionClaim:{contractRevision:"1",objectiveSatisfied:true,criteria:[{criterionId:"objective",status:"satisfied",evidenceRefs:[]}],remainingWork:[]},evidence:[],verification:[]}.`,
|
||||
"Do not create files, plans, or additional work.",
|
||||
].join("\n"),
|
||||
buildMatchers: (nonce, execution) =>
|
||||
|
|
@ -689,7 +689,7 @@ export const openRouterBreadthTasks: readonly RunnerTaskFixture[] = [
|
|||
`The Plan body must contain ${breadthMarker("P_READY", nonce)}.`,
|
||||
"Call write_document for key `plan`, then call request_human_input exactly once with interactionKind `confirmation`, targetRevisionId set to the returned latest Plan revision, and continuationPolicy `wake_assignee`.",
|
||||
"Do not call paperclip_finish while confirmation is pending.",
|
||||
`After that exact Plan revision is accepted, return ${breadthMarker("P_OK", nonce)} visibly and call paperclip_finish with that marker as the summary.`,
|
||||
`After that exact Plan revision is accepted, return ${breadthMarker("P_OK", nonce)} visibly and call paperclip_finish exactly once with {reportedWorkDisposition:"done",summary:"${breadthMarker("P_OK", nonce)}",completionClaim:{contractRevision:"1",objectiveSatisfied:true,criteria:[{criterionId:"objective",status:"satisfied",evidenceRefs:[]}],remainingWork:[]},evidence:[],verification:[]}.`,
|
||||
"Do not create files, child tasks, or unrelated work.",
|
||||
].join("\n"),
|
||||
buildMatchers: (nonce, execution) =>
|
||||
|
|
|
|||
Loading…
Reference in New Issue