Docs / More
Real-agent benchmark (2.x)
Claude Code with and without Prism on public repos: tokens, cost, time and success.
Prism 2.x real-agent benchmark (B6)
Measured 2026-10-17. Model claude-opus-5-5, opencode/mimo-v2.6-flash-free. 10 minute limit per session, no token cap.
Public repos only: es-toolkit, hono.
Method
- Tasks are real fix commits from the repos above. The fix's tests are applied on the parent commit, where they fail. The agent is told to make them pass without editing tests.
- A session succeeds when those tests pass and the agent left the test files untouched.
- Arm A is the host alone (no MCP servers, project settings only). Arm B is the same host plus the local Prism build (MCP server, hooks, skills). Same model, prompt and limit.
- Each session runs in an isolated HOME. Sessions are shuffled with a fixed seed so arms share time-of-day effects.
- Tokens are input + output + cache read + cache write, summed over every API call in the session transcript (each message counted once, subagents included). Cost is the host's reported session total (API-equivalent list price; this run used a subscription sign-in, so nothing was billed per token).
- Medians with interquartile range and min–max. Raw per-session data:
2.x.json.
Claude Code
| Without Prism (A) | With Prism (B) | |
|---|---|---|
| Sessions | 22 | 22 |
| Success | 100% | 100% |
| Tokens, median | 229,880 (IQR 152,092–327,668; 92,794–836,593) | 252,821 (IQR 157,236–286,457; 121,477–787,903) |
| Cost, median | $0.241 (IQR $0.182–$0.284; $0.129–$0.606) | $0.246 (IQR $0.177–$0.283; $0.151–$0.662) |
| Wall time, median | 66 s (IQR 42 s–110 s; 25 s–223 s) | 73 s (IQR 47 s–105 s; 22 s–287 s) |
With / without, medians: tokens 1.10×, cost 1.02×.
| Behaviour (median per session, from transcripts) | A | B |
|---|---|---|
| API calls | 9 | 9 |
| Tool calls | 8 | 8 |
| File reads / searches | 1 | 2 |
| Prism MCP calls | – | 0 (0/22 sessions used any) |
| Context injected by host config (chars: hooks, skills, MCP instructions) | 7,477 | 10,440 |
Value-line calibration (P13)
Fitted on 6 tasks: NaN tokens per file read without Prism. Held-out 5 tasks: estimated saving NaN vs measured -77,021 (NaN%). Fail: fitted tokens-per-file is not positive: arm B did not save tokens.
opencode
| Without Prism (A) | With Prism (B) | |
|---|---|---|
| Sessions | 11 | 11 |
| Success | 73% | 73% |
| Tokens, median | 574,282 (IQR 492,199–721,295; 188,964–1,391,775) | 414,169 (IQR 328,148–543,960; 240,143–1,030,587) |
| Cost, median | $0.000 (IQR $0.000–$0.000; $0.000–$0.000) | $0.000 (IQR $0.000–$0.000; $0.000–$0.000) |
| Wall time, median | 512 s (IQR 345 s–571 s; 109 s–600 s) | 401 s (IQR 276 s–567 s; 99 s–600 s) |
With / without, medians: tokens 0.72×, cost –.
| Behaviour (median per session, from transcripts) | A | B |
|---|---|---|
| API calls | 0 | 0 |
| Tool calls | 24 | 19 |
| File reads / searches | 0 | 0 |
| Prism MCP calls | – | 2 (11/11 sessions used any) |
| Context injected by host config (chars: hooks, skills, MCP instructions) | NaN | NaN |
Value-line calibration (P13)
Not calibrated: Prism sent no files in arm B, nothing to calibrate.
Per-task medians: 2.x per task.