HarnessFacts
mcp.stdio / v1.0.0

MCP stdio

Measures actual invocation of a deterministic get_token MCP tool. A server-side receipt and matching output are both required. HTTP uses a loopback-only Streamable HTTP endpoint.

What counts as a pass?

The evaluator checks tool-call-and-file against server-generated token. Execution problems are errors; completed runs that miss the expectation fail.

Run this test

harnessfacts run codex mcp.stdio

Exact prompt

Call the get_token tool on the harnessfacts MCP server, then write its returned token exactly into result.txt.

Exact fixture

task.txt
Call the get_token tool on the harnessfacts MCP server, then write its returned token exactly into result.txt.

Adapters map instruction and skill files to the product's native paths. Inspect the adapters ↗

Evaluator source
export { evaluate } from "../evaluate.ts";

import { setupMcp } from "../setup.ts";
export const setup = (base: string) => setupMcp(base, "stdio");

All recorded runs

AgentVersionEnvironmentResult / evidence
antigravity-cli1.1.27linux x64pass →
codex0.153.4linux x64fail →
codex0.153.4linux x64pass →
codex0.153.4linux x64pass →
codex0.153.4linux x64fail →
opencode1.18.29linux x64pass →
opencode1.18.29linux x64pass →
opencode1.18.29linux x64pass →