execution.structured-output / v1.0.0
Structured output
Measures whether the requested machine-readable response contains the expected token. Adapters extract only final response payloads from documented JSON envelopes.
What counts as a pass?
The evaluator checks json-object against STRUCTURED. Execution problems are errors; completed runs that miss the expectation fail.
Run this test
harnessfacts run codex execution.structured-outputExact prompt
Respond with a JSON object containing only the property token with value STRUCTURED. Do not include prose or code fences.
Exact fixture
task.txt
Respond with a JSON object containing only the property token with value STRUCTURED. Do not include prose or code fences.
Adapters map instruction and skill files to the product's native paths. Inspect the adapters ↗
Evaluator source
import type { EvaluationContext } from "../../../packages/core/src/types.ts";
export async function evaluate(ctx: EvaluationContext) {
let pass = false;
try {
const value = JSON.parse((await ctx.read(".response.json")) || "null");
pass =
value?.token === ctx.test.expected.value &&
Object.keys(value).length === 1;
} catch {
/* Invalid JSON fails the capability test. */
}
return {
pass,
summary: pass
? "Structured response matched the expected object."
: "No valid structured final response.",
};
}