Methodology

HarnessFacts records what an agent demonstrably did in a specific test, version, environment, and configuration. It is a compatibility notebook, not an intelligence benchmark.

Documented, observed, failed, unknown

• Documented: an official source describes the feature. Claims are stored in claims/, with a source URL and access date. Documentation never creates an observation.
• Observed: an executed test matched its deterministic expectation (pass).
• Failed: the agent process completed, but the expected behavior was not observed (fail).
• Unknown: no sufficient completed observation. Missing data is never treated as failed or unsupported.

Errors mean execution or evaluation could not establish the capability. Skipped means the harness intentionally did not run it, including an unimplemented adapter mapping. Unsupported is reserved for an explicitly established missing interaction mode; it is not inferred from missing data. Documentation contradictions stay visible.

What PASS means

The recorded execution matched a small exact expectation: for example, a token written in a file. The evaluator uses no LLM judgment. MCP tests require a server-side tool receipt and matching output. Structured-output tests parse the final response payload, not text found anywhere in an event stream.

PASS does not establish universal or future-version support, security, intelligence, quality, reliability, or platform portability. A successful scope-boundary test would not be a security guarantee.

The executable suite currently groups tests into instructions, skills, MCP, execution, and Git behavior. Exit-code tests use one successful task and one deterministic failure script: an unreached failure is an execution error, a reached failure with exit code zero fails, and a reached failure with a non-zero exit passes. Worktree and scope tests use disposable Git fixtures and report observed file behavior without making security claims. Hooks remain unknown until a supported agent exposes a deterministic noninteractive lifecycle configuration.

Versions and environments

Every result records exact agent and test versions, OS and OS version, architecture, Bun version, timestamp, duration, process exit status, and provenance. Agent logs describe the selected model where the CLI supplies it. Models may change with an agent’s defaults; results are observations of that complete setup, not controlled model comparisons.

Instruction fixtures are mapped to each product’s native filenames by the adapter. The prompt and token evaluator remain the same. Aider’s root instruction setup uses its native read configuration and is not evidence of automatic AGENTS.md discovery.

Reading the matrix

The latest index groups runs by agent ID/version, test ID/version, and full recorded environment. All history is retained. A compact capability cell shows the most recent completed observation, with its version and OS; agent pages show every recorded attempt. A newer error is shown separately and cannot silently erase a prior observation. Tests without an executable implementation remain unknown.

A repeated run counts as another observation, not another covered configuration. The project does not calculate an overall winner or score. Check the run date, exact environment, failure history, and documentation before drawing a conclusion.

Provenance and evidence

Local runs default to community. Official means a maintainer-controlled environment; neither value proves authenticity or provides a signature. Evidence includes stdout, stderr, inspected artifacts, and SHA-256 hashes. Hashes are computed after redaction and must match the committed bytes. A validator verifies integrity, not truthfulness of an author’s claims.

No results from the mock adapters in project tests enter the published dataset. The initial results were generated by real installed CLIs. Runs use temporary fixtures and homes, sanitized environments, timeouts, and bounded capture. These measures are not a sandbox. Review evidence before publishing it.