execution.exit-failure / v1.0.0
Reached task failure exit
Measures whether a deterministic reached task failure is communicated through a non-zero process exit code.
What counts as a pass?
The evaluator checks exit-code against non-zero. Execution problems are errors; completed runs that miss the expectation fail.
Run this test
harnessfacts run codex execution.exit-failureExact prompt
Run `sh fail-task.sh` as the final action for this task. The script intentionally writes EXIT_FAILURE_REACHED and exits with a non-zero status. Do not recover from or hide that failure; the harness is measuring whether it reaches the CLI.
Exact fixture
fail-task.sh
#!/bin/sh printf 'EXIT_FAILURE_REACHED\n' > result.txt exit 17
task.txt
Run the deterministic failure script described in the prompt.
Adapters map instruction and skill files to the product's native paths. Inspect the adapters ↗
Evaluator source
import type { EvaluationContext } from "../../../packages/core/src/types.ts";
export async function evaluate(ctx: EvaluationContext) {
const reached =
(await ctx.read("result.txt"))?.trim() === "EXIT_FAILURE_REACHED";
if (!reached)
return {
pass: false,
error: true,
summary: "The deterministic failing task was not reached.",
};
const pass = ctx.output.exitCode !== null && ctx.output.exitCode !== 0;
return {
pass,
summary: pass
? "Reached task failure returned a non-zero exit code."
: "Reached task failure returned exit code 0.",
};
}