feat(runner): validate structured question responses (#12420)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Rust runner now owns a bounded ACPX session and a fail-closed turn lifecycle > - Provider questions pause a turn and must return structured answers to the same persisted question set > - JSON Schema validates the wire shape, but it cannot validate identifiers and constraints across two documents > - Unknown questions, invalid choices, and malformed custom answers must fail before any provider receives them > - This pull request adds only the package-local response validator and tests > - The benefit is a small trust boundary that later request-resolution code can use without changing production selection ## Linked Issues or Issue Description Refs #12419 ## What Changed - Validate `paperclip.question_response.v1` against its versioned JSON Schema. - Bound serialized responses to 768 KiB before validation. - Require answer identifiers to match the exact persisted question set. - Require answers for required questions and reject unknown question identifiers. - Enforce text, single-select, and multi-select answer modes. - Match the existing TypeScript numeric syntax, including decimal, exponent, hexadecimal, octal, and binary input. - Match ECMAScript trimming exactly, including BOM whitespace while rejecting Unicode NEL rather than inheriting Rust-specific whitespace behavior. - Enforce known options, custom-answer policy, text length, pattern, and numeric constraints. - Validate duplicate option IDs, inverted bounds, and dynamic patterns before answer lookup so malformed optional questions fail closed even when unanswered. - Match JavaScript UTF-16 code-unit length semantics for text constraints and the 100,000-unit response-field bound. - Preserve the public optional `recommended` question-option field in the versioned schema, generated schema bundle, and Rust validation path. - Return typed validation errors for malformed inputs without panics. - Export the validator from the Rust runner core. - Add table-driven tests for valid, mismatched, malformed, oversized, and numeric-boundary responses. - Document the package-local structured-response boundary. - Add `num-bigint` 0.4 and `num-traits` 0.2 as direct runner-core dependencies for exact arbitrary-length radix parsing and one-step JavaScript Number rounding; update only the package-local runner Cargo lockfile. - Do not change the repository PNPM lockfile, workflows, runnerd selection, server behavior, UI, or migrations. ## Verification - Replay base: `9a9fdf06ee4142f77427db30efccc4c43056f64b` (`master` after #12419 merged). - Exact replay head: `fad92b3fb348b66ddb10dde44b7b060e55c4fe96`. - Stable patch ID: `4d6ffbd519dd081f7ea530977cd965bd4569fc75`; this is the prepared two-commit delta plus the focused cross-language parity fix found during replay review. - The exact delta is 10 files, 712 additions, and 2 deletions, all in `packages/paperclip-runner`. - The package-local `packages/paperclip-runner/runner/Cargo.lock` records the two direct runner-core dependencies; their already-resolved versions and checksums are unchanged. - The question-set schema source, generated TypeScript schema bundle, and protocol manifest hash are updated together; the schema SHA-256 is `42b5441a3d388851dacb6e4500dfd4a17d878eded2e724228078b647e7440d3f`. - GitHub Actions run `33366812025`, attempt 2: **PASSED** on the exact replay head (23/23 jobs passed; a failed-job-only retry cleared one unrelated ACPX runtime-host timeout). - Greptile: **5/5** on the exact replay head with zero unresolved review threads; Superagent, Socket, Snyk, and contributor-trust checks also passed. - No local test result is claimed. GitHub Actions is the authoritative verification environment for this replayed revision. ## Risks - The validator compiles the embedded response schema for each submission. Responses are user-paced and bounded, so this keeps the slice simple without affecting a hot event path. - The persisted question set is the source of truth for identifiers and constraints. A malformed persisted set fails closed. - Numeric input follows the existing structured-question contract, including JavaScript-prefixed syntax. Optional whitespace-only answers are rejected instead of being treated as an omitted value. - Error messages identify the invalid field but do not include answer text. - No production path invokes this validator in this pull request. Request resolution remains the next slice. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex with GPT-5.6, agentic reasoning, tool use, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge
This commit is contained in:
Dotta
GitHub
parent
9a9fdf06ee
commit
3aa2065d08
|
|
@ -103,6 +103,10 @@ then resumes the same verified persistent identity in a fresh generation. This
|
|||
prevents a late session-lifetime MCP callback from inheriting the next turn's
|
||||
event authority.
|
||||
|
||||
The Rust question-response validator checks the versioned response envelope
|
||||
against the exact persisted question IDs, answer modes, options, required
|
||||
answers, custom-answer policy, and text constraints before provider delivery.
|
||||
|
||||
Run the complete contract gate with:
|
||||
|
||||
```sh
|
||||
|
|
|
|||
|
|
@ -65,7 +65,7 @@
|
|||
{
|
||||
"path": "schemas/question-set.schema.json",
|
||||
"id": "https://paperclip.dev/schemas/prp/v1/question-set.schema.json",
|
||||
"sha256": "3bda40311ae3153bbf9d4a8c59f716680e17d711a4db3570193963918359751c"
|
||||
"sha256": "42b5441a3d388851dacb6e4500dfd4a17d878eded2e724228078b647e7440d3f"
|
||||
},
|
||||
{
|
||||
"path": "schemas/request.schema.json",
|
||||
|
|
|
|||
|
|
@ -23,7 +23,8 @@
|
|||
"properties": {
|
||||
"id": { "type": "string", "minLength": 1, "maxLength": 160 },
|
||||
"label": { "type": "string", "minLength": 1, "maxLength": 1000 },
|
||||
"description": { "type": "string", "maxLength": 4000 }
|
||||
"description": { "type": "string", "maxLength": 4000 },
|
||||
"recommended": { "type": "boolean" }
|
||||
},
|
||||
"additionalProperties": false
|
||||
},
|
||||
|
|
|
|||
|
|
@ -663,6 +663,8 @@ dependencies = [
|
|||
"getrandom 0.3.4",
|
||||
"hmac",
|
||||
"jsonschema",
|
||||
"num-bigint",
|
||||
"num-traits",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"sha2",
|
||||
|
|
|
|||
|
|
@ -13,6 +13,8 @@ aes-gcm = "0.10"
|
|||
getrandom = "0.3"
|
||||
hmac = "0.12"
|
||||
jsonschema = { version = "0.50", default-features = false }
|
||||
num-bigint = "0.4"
|
||||
num-traits = "0.2"
|
||||
serde = { version = "1.0", features = ["derive"] }
|
||||
serde_json = "1.0"
|
||||
sha2 = "0.10"
|
||||
|
|
|
|||
|
|
@ -10,6 +10,8 @@ aes-gcm.workspace = true
|
|||
getrandom.workspace = true
|
||||
hmac.workspace = true
|
||||
jsonschema.workspace = true
|
||||
num-bigint.workspace = true
|
||||
num-traits.workspace = true
|
||||
serde.workspace = true
|
||||
serde_json.workspace = true
|
||||
sha2.workspace = true
|
||||
|
|
|
|||
|
|
@ -14,6 +14,7 @@ pub mod process_supervisor;
|
|||
pub mod provider_backend;
|
||||
pub mod provider_bridge;
|
||||
pub mod provider_events;
|
||||
pub mod question_response;
|
||||
pub mod replay;
|
||||
|
||||
use std::error::Error;
|
||||
|
|
|
|||
|
|
@ -0,0 +1,375 @@
|
|||
use std::collections::BTreeSet;
|
||||
|
||||
use num_bigint::BigUint;
|
||||
use num_traits::ToPrimitive;
|
||||
use serde_json::{json, Value};
|
||||
|
||||
use crate::local_runner::LocalRunnerError;
|
||||
|
||||
const MAX_QUESTION_RESPONSE_BYTES: usize = 768 * 1024;
|
||||
const MAX_QUESTION_ANSWER_CODE_UNITS: usize = 100_000;
|
||||
|
||||
/// Validates a provider-neutral response against the exact persisted question
|
||||
/// set that produced it. The JSON Schema owns the wire shape; this function
|
||||
/// adds cross-document identifiers, modes, required answers, and constraints.
|
||||
pub fn validate_question_response(
|
||||
question_set: &Value,
|
||||
response: &Value,
|
||||
) -> Result<(), LocalRunnerError> {
|
||||
let bytes = serde_json::to_vec(response).map_err(|error| {
|
||||
LocalRunnerError::invalid(format!("question response is not serializable: {error}"))
|
||||
})?;
|
||||
if bytes.len() > MAX_QUESTION_RESPONSE_BYTES {
|
||||
return Err(LocalRunnerError::invalid(
|
||||
"question response exceeds its bounded transport contract",
|
||||
));
|
||||
}
|
||||
let question_set_schema: Value = serde_json::from_str(include_str!(
|
||||
"../../../../protocol/schemas/question-set.schema.json"
|
||||
))
|
||||
.map_err(|_| LocalRunnerError::invalid("embedded question-set schema is invalid"))?;
|
||||
let question_set_validator = jsonschema::validator_for(&question_set_schema)
|
||||
.map_err(|_| LocalRunnerError::invalid("embedded question-set schema cannot compile"))?;
|
||||
if !question_set_validator.is_valid(question_set) {
|
||||
return Err(LocalRunnerError::invalid(
|
||||
"persisted question set failed the Paperclip question-set schema",
|
||||
));
|
||||
}
|
||||
let schema: Value = serde_json::from_str(include_str!(
|
||||
"../../../../protocol/schemas/question-response.schema.json"
|
||||
))
|
||||
.map_err(|_| LocalRunnerError::invalid("embedded question-response schema is invalid"))?;
|
||||
let validator = jsonschema::validator_for(&schema).map_err(|_| {
|
||||
LocalRunnerError::invalid("embedded question-response schema cannot compile")
|
||||
})?;
|
||||
if !validator.is_valid(response) {
|
||||
return Err(LocalRunnerError::invalid(
|
||||
"response failed the Paperclip question-response schema",
|
||||
));
|
||||
}
|
||||
|
||||
let questions = question_set
|
||||
.get("questions")
|
||||
.and_then(Value::as_array)
|
||||
.ok_or_else(|| LocalRunnerError::invalid("persisted question set is malformed"))?;
|
||||
let answers = response
|
||||
.get("answers")
|
||||
.and_then(Value::as_object)
|
||||
.ok_or_else(|| LocalRunnerError::invalid("question response answers are malformed"))?;
|
||||
let question_ids = questions
|
||||
.iter()
|
||||
.filter_map(|question| question.get("id").and_then(Value::as_str))
|
||||
.collect::<BTreeSet<_>>();
|
||||
if question_ids.len() != questions.len() {
|
||||
return Err(LocalRunnerError::invalid(
|
||||
"persisted question set has invalid or duplicate ids",
|
||||
));
|
||||
}
|
||||
for question in questions {
|
||||
validate_persisted_question(question)?;
|
||||
}
|
||||
if answers.keys().any(|id| !question_ids.contains(id.as_str())) {
|
||||
return Err(LocalRunnerError::invalid(
|
||||
"question response contains an unknown question id",
|
||||
));
|
||||
}
|
||||
|
||||
for question in questions {
|
||||
let question_id = question
|
||||
.get("id")
|
||||
.and_then(Value::as_str)
|
||||
.ok_or_else(|| LocalRunnerError::invalid("persisted question id is malformed"))?;
|
||||
validate_answer(question, answers.get(question_id))?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn validate_answer(question: &Value, answer: Option<&Value>) -> Result<(), LocalRunnerError> {
|
||||
let question_id = question
|
||||
.get("id")
|
||||
.and_then(Value::as_str)
|
||||
.ok_or_else(|| LocalRunnerError::invalid("persisted question id is malformed"))?;
|
||||
let required = question
|
||||
.get("required")
|
||||
.and_then(Value::as_bool)
|
||||
.ok_or_else(|| LocalRunnerError::invalid("persisted question requirement is malformed"))?;
|
||||
let Some(answer) = answer else {
|
||||
return if required {
|
||||
Err(LocalRunnerError::invalid(format!(
|
||||
"question response is missing required answer {question_id}"
|
||||
)))
|
||||
} else {
|
||||
Ok(())
|
||||
};
|
||||
};
|
||||
let answer = answer
|
||||
.as_object()
|
||||
.ok_or_else(|| LocalRunnerError::invalid("question response answer is malformed"))?;
|
||||
let selected = answer
|
||||
.get("selectedOptionIds")
|
||||
.and_then(Value::as_array)
|
||||
.map(|values| {
|
||||
values
|
||||
.iter()
|
||||
.map(|value| {
|
||||
value.as_str().ok_or_else(|| {
|
||||
LocalRunnerError::invalid("question response option id is malformed")
|
||||
})
|
||||
})
|
||||
.collect::<Result<Vec<_>, _>>()
|
||||
})
|
||||
.transpose()?;
|
||||
let text = answer.get("text").and_then(Value::as_str);
|
||||
let custom = answer.get("customText").and_then(Value::as_str);
|
||||
if [text, custom]
|
||||
.into_iter()
|
||||
.flatten()
|
||||
.any(|value| javascript_string_length(value) > MAX_QUESTION_ANSWER_CODE_UNITS)
|
||||
{
|
||||
return Err(LocalRunnerError::invalid(format!(
|
||||
"question response answer {question_id} exceeds its text bound"
|
||||
)));
|
||||
}
|
||||
let has_value = selected.as_ref().is_some_and(|values| !values.is_empty())
|
||||
|| text.is_some_and(|value| !value.trim_matches(is_ecmascript_whitespace).is_empty())
|
||||
|| custom.is_some_and(|value| !value.trim_matches(is_ecmascript_whitespace).is_empty());
|
||||
if !has_value {
|
||||
return Err(LocalRunnerError::invalid(format!(
|
||||
"question response answer {question_id} is {}",
|
||||
if required { "required" } else { "empty" }
|
||||
)));
|
||||
}
|
||||
|
||||
match question.get("answerMode").and_then(Value::as_str) {
|
||||
Some("text") => {
|
||||
if selected.as_ref().is_some_and(|values| !values.is_empty()) || custom.is_some() {
|
||||
return Err(LocalRunnerError::invalid(format!(
|
||||
"text answer {question_id} cannot contain selection fields"
|
||||
)));
|
||||
}
|
||||
if let Some(text) = text {
|
||||
validate_text_constraints(question_id, question, text)?;
|
||||
}
|
||||
}
|
||||
Some("single_select" | "multi_select") => {
|
||||
if text.is_some() {
|
||||
return Err(LocalRunnerError::invalid(format!(
|
||||
"select answer {question_id} cannot contain text"
|
||||
)));
|
||||
}
|
||||
let allowed = question
|
||||
.get("options")
|
||||
.and_then(Value::as_array)
|
||||
.into_iter()
|
||||
.flatten()
|
||||
.filter_map(|option| option.get("id").and_then(Value::as_str))
|
||||
.collect::<BTreeSet<_>>();
|
||||
if selected
|
||||
.iter()
|
||||
.flatten()
|
||||
.any(|option_id| !allowed.contains(option_id))
|
||||
{
|
||||
return Err(LocalRunnerError::invalid(format!(
|
||||
"select answer {question_id} contains an unknown option"
|
||||
)));
|
||||
}
|
||||
if question.get("answerMode").and_then(Value::as_str) == Some("single_select")
|
||||
&& selected.as_ref().is_some_and(|values| values.len() > 1)
|
||||
{
|
||||
return Err(LocalRunnerError::invalid(format!(
|
||||
"single-select answer {question_id} chose multiple options"
|
||||
)));
|
||||
}
|
||||
if custom.is_some()
|
||||
&& question
|
||||
.pointer("/customAnswer/enabled")
|
||||
.and_then(Value::as_bool)
|
||||
!= Some(true)
|
||||
{
|
||||
return Err(LocalRunnerError::invalid(format!(
|
||||
"select answer {question_id} does not allow custom text"
|
||||
)));
|
||||
}
|
||||
if question.get("answerMode").and_then(Value::as_str) == Some("single_select")
|
||||
&& selected.as_ref().is_some_and(|values| !values.is_empty())
|
||||
&& custom
|
||||
.is_some_and(|value| !value.trim_matches(is_ecmascript_whitespace).is_empty())
|
||||
{
|
||||
return Err(LocalRunnerError::invalid(format!(
|
||||
"single-select answer {question_id} cannot combine an option and custom text"
|
||||
)));
|
||||
}
|
||||
if let Some(custom) = custom {
|
||||
validate_text_constraints(question_id, question, custom)?;
|
||||
}
|
||||
}
|
||||
_ => {
|
||||
return Err(LocalRunnerError::invalid(
|
||||
"persisted question answer mode is malformed",
|
||||
))
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn validate_text_constraints(
|
||||
question_id: &str,
|
||||
question: &Value,
|
||||
text: &str,
|
||||
) -> Result<(), LocalRunnerError> {
|
||||
let Some(validation) = question.get("textValidation") else {
|
||||
return Ok(());
|
||||
};
|
||||
let length = javascript_string_length(text) as u64;
|
||||
if validation
|
||||
.get("minLength")
|
||||
.and_then(Value::as_u64)
|
||||
.is_some_and(|minimum| length < minimum)
|
||||
|| validation
|
||||
.get("maxLength")
|
||||
.and_then(Value::as_u64)
|
||||
.is_some_and(|maximum| length > maximum)
|
||||
{
|
||||
return Err(LocalRunnerError::invalid(format!(
|
||||
"answer {question_id} violates its text length constraint"
|
||||
)));
|
||||
}
|
||||
if let Some(pattern) = validation.get("pattern").and_then(Value::as_str) {
|
||||
let pattern_schema = json!({"type":"string","pattern":pattern});
|
||||
let validator = jsonschema::validator_for(&pattern_schema)
|
||||
.map_err(|_| LocalRunnerError::invalid("persisted question pattern cannot compile"))?;
|
||||
if !validator.is_valid(&Value::String(text.to_owned())) {
|
||||
return Err(LocalRunnerError::invalid(format!(
|
||||
"answer {question_id} does not match its required pattern"
|
||||
)));
|
||||
}
|
||||
}
|
||||
if matches!(
|
||||
validation.get("inputType").and_then(Value::as_str),
|
||||
Some("number" | "integer")
|
||||
) {
|
||||
let number = parse_javascript_number(text).ok_or_else(|| {
|
||||
LocalRunnerError::invalid(format!("answer {question_id} must be numeric"))
|
||||
})?;
|
||||
if !number.is_finite()
|
||||
|| (validation.get("inputType").and_then(Value::as_str) == Some("integer")
|
||||
&& number.fract() != 0.0)
|
||||
|| validation
|
||||
.get("minimum")
|
||||
.and_then(Value::as_f64)
|
||||
.is_some_and(|minimum| number < minimum)
|
||||
|| validation
|
||||
.get("maximum")
|
||||
.and_then(Value::as_f64)
|
||||
.is_some_and(|maximum| number > maximum)
|
||||
{
|
||||
return Err(LocalRunnerError::invalid(format!(
|
||||
"answer {question_id} violates its numeric constraint"
|
||||
)));
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn validate_persisted_question(question: &Value) -> Result<(), LocalRunnerError> {
|
||||
let question_id = question
|
||||
.get("id")
|
||||
.and_then(Value::as_str)
|
||||
.ok_or_else(|| LocalRunnerError::invalid("persisted question id is malformed"))?;
|
||||
if let Some(options) = question.get("options").and_then(Value::as_array) {
|
||||
let option_ids = options
|
||||
.iter()
|
||||
.filter_map(|option| option.get("id").and_then(Value::as_str))
|
||||
.collect::<BTreeSet<_>>();
|
||||
if option_ids.len() != options.len() {
|
||||
return Err(LocalRunnerError::invalid(format!(
|
||||
"persisted question {question_id} has invalid or duplicate option ids"
|
||||
)));
|
||||
}
|
||||
}
|
||||
let Some(validation) = question.get("textValidation") else {
|
||||
return Ok(());
|
||||
};
|
||||
if validation
|
||||
.get("minLength")
|
||||
.and_then(Value::as_u64)
|
||||
.zip(validation.get("maxLength").and_then(Value::as_u64))
|
||||
.is_some_and(|(minimum, maximum)| minimum > maximum)
|
||||
{
|
||||
return Err(LocalRunnerError::invalid(format!(
|
||||
"persisted question {question_id} has inverted text length bounds"
|
||||
)));
|
||||
}
|
||||
if validation
|
||||
.get("minimum")
|
||||
.and_then(Value::as_f64)
|
||||
.zip(validation.get("maximum").and_then(Value::as_f64))
|
||||
.is_some_and(|(minimum, maximum)| minimum > maximum)
|
||||
{
|
||||
return Err(LocalRunnerError::invalid(format!(
|
||||
"persisted question {question_id} has inverted numeric bounds"
|
||||
)));
|
||||
}
|
||||
if let Some(pattern) = validation.get("pattern").and_then(Value::as_str) {
|
||||
let pattern_schema = json!({"type":"string","pattern":pattern});
|
||||
jsonschema::validator_for(&pattern_schema).map_err(|_| {
|
||||
LocalRunnerError::invalid(format!(
|
||||
"persisted question {question_id} has an invalid text pattern"
|
||||
))
|
||||
})?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn javascript_string_length(value: &str) -> usize {
|
||||
value.encode_utf16().count()
|
||||
}
|
||||
|
||||
fn parse_javascript_number(value: &str) -> Option<f64> {
|
||||
let value = value.trim_matches(is_ecmascript_whitespace);
|
||||
if value.is_empty() {
|
||||
return Some(0.0);
|
||||
}
|
||||
for (prefixes, radix) in [(["0x", "0X"], 16), (["0o", "0O"], 8), (["0b", "0B"], 2)] {
|
||||
if let Some(digits) = prefixes
|
||||
.iter()
|
||||
.find_map(|prefix| value.strip_prefix(prefix))
|
||||
{
|
||||
if digits.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let digits = digits
|
||||
.chars()
|
||||
.map(|character| character.to_digit(radix).map(|digit| digit as u8))
|
||||
.collect::<Option<Vec<_>>>()?;
|
||||
// JavaScript parses the entire prefixed integer exactly and rounds
|
||||
// once when converting it to Number. BigUint's f64 conversion uses
|
||||
// round-to-odd before the final nearest-ties-to-even conversion,
|
||||
// preserving that behavior without intermediate digit rounding.
|
||||
return BigUint::from_radix_be(&digits, radix).and_then(|number| number.to_f64());
|
||||
}
|
||||
}
|
||||
value.parse::<f64>().ok()
|
||||
}
|
||||
|
||||
fn is_ecmascript_whitespace(character: char) -> bool {
|
||||
matches!(
|
||||
character,
|
||||
'\u{0009}'
|
||||
| '\u{000a}'
|
||||
| '\u{000b}'
|
||||
| '\u{000c}'
|
||||
| '\u{000d}'
|
||||
| '\u{0020}'
|
||||
| '\u{00a0}'
|
||||
| '\u{1680}'
|
||||
| '\u{2000}'
|
||||
..='\u{200a}'
|
||||
| '\u{2028}'
|
||||
| '\u{2029}'
|
||||
| '\u{202f}'
|
||||
| '\u{205f}'
|
||||
| '\u{3000}'
|
||||
| '\u{feff}'
|
||||
)
|
||||
}
|
||||
|
|
@ -0,0 +1,320 @@
|
|||
use num_bigint::BigUint;
|
||||
use paperclip_runner_core::question_response::validate_question_response;
|
||||
use serde_json::{json, Value};
|
||||
|
||||
fn question_set() -> Value {
|
||||
json!({
|
||||
"schema":"paperclip.question_set.v1",
|
||||
"questions":[
|
||||
{
|
||||
"id":"target",
|
||||
"prompt":"Which target?",
|
||||
"required":true,
|
||||
"answerMode":"single_select",
|
||||
"options":[{"id":"first","label":"First","recommended":true},{"id":"second","label":"Second"}],
|
||||
"customAnswer":{"enabled":true}
|
||||
},
|
||||
{
|
||||
"id":"regions",
|
||||
"prompt":"Which regions?",
|
||||
"required":false,
|
||||
"answerMode":"multi_select",
|
||||
"options":[{"id":"east","label":"East"},{"id":"west","label":"West"}]
|
||||
},
|
||||
{
|
||||
"id":"notes",
|
||||
"prompt":"Add notes",
|
||||
"required":true,
|
||||
"answerMode":"text",
|
||||
"textValidation":{"minLength":2,"maxLength":5,"pattern":"^[A-Z]+$"}
|
||||
},
|
||||
{
|
||||
"id":"count",
|
||||
"prompt":"How many?",
|
||||
"required":false,
|
||||
"answerMode":"text",
|
||||
"textValidation":{"inputType":"integer","minimum":1,"maximum":3}
|
||||
}
|
||||
]
|
||||
})
|
||||
}
|
||||
|
||||
fn valid_response() -> Value {
|
||||
json!({
|
||||
"schema":"paperclip.question_response.v1",
|
||||
"answers":{
|
||||
"target":{"selectedOptionIds":["first"]},
|
||||
"regions":{"selectedOptionIds":["east","west"]},
|
||||
"notes":{"text":"YES"},
|
||||
"count":{"text":"2"}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn accepts_answers_that_match_the_exact_question_set() {
|
||||
validate_question_response(&question_set(), &valid_response()).unwrap();
|
||||
|
||||
let mut custom = valid_response();
|
||||
custom["answers"]["target"] = json!({"customText":"another"});
|
||||
validate_question_response(&question_set(), &custom).unwrap();
|
||||
|
||||
let mut javascript_numeric_syntax = valid_response();
|
||||
javascript_numeric_syntax["answers"]["count"] = json!({"text":"\u{feff}0x2\u{feff}"});
|
||||
validate_question_response(&question_set(), &javascript_numeric_syntax).unwrap();
|
||||
|
||||
let mut empty_custom_with_selection = valid_response();
|
||||
empty_custom_with_selection["answers"]["target"] =
|
||||
json!({"selectedOptionIds":["first"],"customText":"\u{feff}"});
|
||||
validate_question_response(&question_set(), &empty_custom_with_selection).unwrap();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rounds_large_prefixed_integers_like_javascript_number() {
|
||||
let mut bounded_set = question_set();
|
||||
bounded_set["questions"][3]["textValidation"]["minimum"] = json!(1_152_921_504_606_847_200_u64);
|
||||
bounded_set["questions"][3]["textValidation"]["maximum"] = json!(1_152_921_504_606_847_200_u64);
|
||||
|
||||
for value in [
|
||||
"0x1000000000000081",
|
||||
"0o100000000000000000201",
|
||||
"0b1000000000000000000000000000000000000000000000000000010000001",
|
||||
] {
|
||||
let mut response = valid_response();
|
||||
response["answers"]["count"] = json!({"text":value});
|
||||
validate_question_response(&bounded_set, &response)
|
||||
.unwrap_or_else(|error| panic!("{value} should match JavaScript Number: {error}"));
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn matches_javascript_radix_overflow_midpoint() {
|
||||
let mut unbounded_set = question_set();
|
||||
let validation = unbounded_set["questions"][3]["textValidation"]
|
||||
.as_object_mut()
|
||||
.unwrap();
|
||||
validation.remove("minimum");
|
||||
validation.remove("maximum");
|
||||
|
||||
let overflow = BigUint::from(1_u8) << 1024_usize;
|
||||
// Number.MAX_VALUE is 2^1024 - 2^971. The midpoint to the
|
||||
// non-representable 2^1024 sentinel is 2^1024 - 2^970. At the midpoint,
|
||||
// nearest-ties-to-even selects the sentinel, which JavaScript exposes as
|
||||
// Infinity; the immediately preceding integer still rounds to MAX_VALUE.
|
||||
let infinite_midpoint = &overflow - (BigUint::from(1_u8) << 970_usize);
|
||||
let largest_finite = &infinite_midpoint - BigUint::from(1_u8);
|
||||
let below_overflow_but_infinite = &overflow - BigUint::from(1_u8);
|
||||
|
||||
for (prefix, radix) in [("0x", 16), ("0o", 8), ("0b", 2)] {
|
||||
let mut response = valid_response();
|
||||
response["answers"]["count"] =
|
||||
json!({"text":format!("{prefix}{}", largest_finite.to_str_radix(radix))});
|
||||
validate_question_response(&unbounded_set, &response).unwrap_or_else(|error| {
|
||||
panic!("the largest finite-rounding base-{radix} integer was rejected: {error}")
|
||||
});
|
||||
|
||||
for value in [&infinite_midpoint, &below_overflow_but_infinite, &overflow] {
|
||||
response["answers"]["count"] =
|
||||
json!({"text":format!("{prefix}{}", value.to_str_radix(radix))});
|
||||
assert!(
|
||||
validate_question_response(&unbounded_set, &response).is_err(),
|
||||
"base-{radix} value that rounds to Infinity was accepted"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn treats_ecmascript_bom_whitespace_as_an_empty_required_answer() {
|
||||
let mut unconstrained_set = question_set();
|
||||
unconstrained_set["questions"][2]
|
||||
.as_object_mut()
|
||||
.unwrap()
|
||||
.remove("textValidation");
|
||||
|
||||
let mut bom_text = valid_response();
|
||||
bom_text["answers"]["notes"] = json!({"text":"\u{feff}"});
|
||||
assert!(validate_question_response(&unconstrained_set, &bom_text).is_err());
|
||||
|
||||
let mut bom_custom = valid_response();
|
||||
bom_custom["answers"]["target"] = json!({"customText":"\u{feff}"});
|
||||
assert!(validate_question_response(&question_set(), &bom_custom).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_present_empty_optional_answers() {
|
||||
let mut optional_text_set = question_set();
|
||||
optional_text_set["questions"][2]["required"] = json!(false);
|
||||
optional_text_set["questions"][2]
|
||||
.as_object_mut()
|
||||
.unwrap()
|
||||
.remove("textValidation");
|
||||
let mut empty_text = valid_response();
|
||||
empty_text["answers"]["notes"] = json!({"text":"\u{feff}\u{2009}"});
|
||||
assert!(validate_question_response(&optional_text_set, &empty_text).is_err());
|
||||
|
||||
let mut optional_custom_set = question_set();
|
||||
optional_custom_set["questions"][0]["required"] = json!(false);
|
||||
let mut empty_custom = valid_response();
|
||||
empty_custom["answers"]["target"] = json!({"customText":"\u{feff}\u{2009}"});
|
||||
assert!(validate_question_response(&optional_custom_set, &empty_custom).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_malformed_persisted_text_constraints() {
|
||||
let malformed_constraints = [
|
||||
json!("not-an-object"),
|
||||
json!({"minLength":"2"}),
|
||||
json!({"maxLength":2.5}),
|
||||
json!({"pattern":false}),
|
||||
json!({"inputType":false}),
|
||||
json!({"minimum":"1"}),
|
||||
json!({"maximum":{}}),
|
||||
];
|
||||
for constraint in malformed_constraints {
|
||||
let mut malformed_set = question_set();
|
||||
malformed_set["questions"][2]["textValidation"] = constraint.clone();
|
||||
assert!(
|
||||
validate_question_response(&malformed_set, &valid_response()).is_err(),
|
||||
"malformed text constraint unexpectedly passed: {constraint}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_semantically_malformed_unanswered_questions() {
|
||||
let cases = [
|
||||
("duplicate option ids", {
|
||||
let mut value = question_set();
|
||||
value["questions"][0]["options"][1]["id"] = json!("first");
|
||||
value
|
||||
}),
|
||||
("inverted text length bounds", {
|
||||
let mut value = question_set();
|
||||
value["questions"][2]["textValidation"]["minLength"] = json!(6);
|
||||
value
|
||||
}),
|
||||
("inverted numeric bounds", {
|
||||
let mut value = question_set();
|
||||
value["questions"][3]["textValidation"]["minimum"] = json!(4);
|
||||
value
|
||||
}),
|
||||
("invalid text pattern", {
|
||||
let mut value = question_set();
|
||||
value["questions"][2]["textValidation"]["pattern"] = json!("[");
|
||||
value
|
||||
}),
|
||||
];
|
||||
let response = json!({"schema":"paperclip.question_response.v1","answers":{}});
|
||||
for (label, mut set) in cases {
|
||||
for question in set["questions"].as_array_mut().unwrap() {
|
||||
question["required"] = json!(false);
|
||||
}
|
||||
assert!(
|
||||
validate_question_response(&set, &response).is_err(),
|
||||
"{label} unexpectedly passed without an answer"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn counts_text_lengths_like_javascript_utf16() {
|
||||
let mut bounded_set = question_set();
|
||||
bounded_set["questions"][2]["textValidation"] = json!({"minLength":2,"maxLength":2});
|
||||
let mut response = valid_response();
|
||||
response["answers"]["notes"] = json!({"text":"😀"});
|
||||
validate_question_response(&bounded_set, &response).unwrap();
|
||||
|
||||
bounded_set["questions"][2]["textValidation"] = json!({"maxLength":1});
|
||||
assert!(validate_question_response(&bounded_set, &response).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_cross_document_and_answer_mode_mismatches() {
|
||||
let cases = [
|
||||
("missing required", {
|
||||
let mut value = valid_response();
|
||||
value["answers"].as_object_mut().unwrap().remove("target");
|
||||
value
|
||||
}),
|
||||
("unknown question", {
|
||||
let mut value = valid_response();
|
||||
value["answers"]["other"] = json!({"text":"x"});
|
||||
value
|
||||
}),
|
||||
("unknown option", {
|
||||
let mut value = valid_response();
|
||||
value["answers"]["target"] = json!({"selectedOptionIds":["other"]});
|
||||
value
|
||||
}),
|
||||
("multiple single selections", {
|
||||
let mut value = valid_response();
|
||||
value["answers"]["target"] = json!({"selectedOptionIds":["first","second"]});
|
||||
value
|
||||
}),
|
||||
("combined single selection", {
|
||||
let mut value = valid_response();
|
||||
value["answers"]["target"] =
|
||||
json!({"selectedOptionIds":["first"],"customText":"other"});
|
||||
value
|
||||
}),
|
||||
("selection on text", {
|
||||
let mut value = valid_response();
|
||||
value["answers"]["notes"] = json!({"selectedOptionIds":["first"]});
|
||||
value
|
||||
}),
|
||||
("pattern mismatch", {
|
||||
let mut value = valid_response();
|
||||
value["answers"]["notes"] = json!({"text":"no"});
|
||||
value
|
||||
}),
|
||||
("numeric mismatch", {
|
||||
let mut value = valid_response();
|
||||
value["answers"]["count"] = json!({"text":"4"});
|
||||
value
|
||||
}),
|
||||
("invalid numeric syntax", {
|
||||
let mut value = valid_response();
|
||||
value["answers"]["count"] = json!({"text":"0xGG"});
|
||||
value
|
||||
}),
|
||||
("non-ECMAScript numeric whitespace", {
|
||||
let mut value = valid_response();
|
||||
value["answers"]["count"] = json!({"text":"\u{0085}2\u{0085}"});
|
||||
value
|
||||
}),
|
||||
];
|
||||
for (label, response) in cases {
|
||||
assert!(
|
||||
validate_question_response(&question_set(), &response).is_err(),
|
||||
"{label} unexpectedly passed"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_malformed_or_oversized_response_envelopes() {
|
||||
assert!(validate_question_response(
|
||||
&question_set(),
|
||||
&json!({"schema":"paperclip.question_response.v2","answers":{}})
|
||||
)
|
||||
.is_err());
|
||||
assert!(validate_question_response(
|
||||
&question_set(),
|
||||
&json!({
|
||||
"schema":"paperclip.question_response.v1",
|
||||
"answers":{"notes":{"text":"x".repeat(800_000)}}
|
||||
})
|
||||
)
|
||||
.is_err());
|
||||
let mut unconstrained_set = question_set();
|
||||
unconstrained_set["questions"][2]
|
||||
.as_object_mut()
|
||||
.unwrap()
|
||||
.remove("textValidation");
|
||||
let mut code_unit_bounded = valid_response();
|
||||
code_unit_bounded["answers"]["notes"] = json!({"text":"😀".repeat(50_000)});
|
||||
validate_question_response(&unconstrained_set, &code_unit_bounded).unwrap();
|
||||
code_unit_bounded["answers"]["notes"] = json!({"text":"😀".repeat(50_001)});
|
||||
assert!(validate_question_response(&unconstrained_set, &code_unit_bounded).is_err());
|
||||
}
|
||||
|
|
@ -2439,6 +2439,9 @@ export const questionSetSchema = {
|
|||
"description": {
|
||||
"type": "string",
|
||||
"maxLength": 4000
|
||||
},
|
||||
"recommended": {
|
||||
"type": "boolean"
|
||||
}
|
||||
},
|
||||
"additionalProperties": false
|
||||
|
|
|
|||
Loading…
Reference in New Issue