Benchmark Status Full report →
Interim snapshot · generated 2026-10-12 19:38 PT

Agentic Framework Benchmark: Status

Interim snapshot, generated 2026-10-12 19:38 PT. Every number below is computed from _bench/data/dataset.json. The matrix is still running, so rankings can move.

INTERIMSame model, same harnessHidden gradersCanonical attempt per run
✓
675graded runs (clean)
OpusHaiku
◆
376Opus 5.5 runs
◇
299Haiku 4.5 runs
▦
40arms (frameworks, controls, apps)
⚑
11tasks with hidden graders
01The story so far

Key takeaways so far

These takeaways are generated from the live data on every rebuild, so they always match the tables below. Interim: cells have 1–3 trials.

    02Spotlight

    Meterless spotlight priority

    Meterless is tested in the ways people actually use it: as a proxy / context layer (World State, H-MEM and Markovian Reasoning through the reference engines), as its own app built on the engines (raw output and after refinement), and as the vibe-development prompt cards, with or without the native audit/remediation step. "Base, same tasks" is the same-model baseline averaged over exactly the tasks that arm has covered.

    The Meterless runs still pending are running now (Opus and Haiku batches). This table refreshes on each collection cycle.

    03Ranking 1

    Ranking 1: frameworks on Opus 5.5 (tasks A–H)

    The composite is pre-registered: 40% quality, 20% reliability, 15% (100 − drift), 15% verification and 10% normalised quality-per-dollar. Relative cost compares each arm with the baseline's cost on the same tasks. Click a column header to sort.

    Lift is the task-balanced mean difference from the baseline, with a stratified-bootstrap 95% CI. With the baseline at about 100, quality lift is capped near 0 by construction.

    04Ranking 2

    Ranking 2: lift on Haiku 4.5 (weaker model, more headroom)

    Partial coverage: compare each lift against ctl-minimal (a short generic instruction) and ctl-placebo. A framework only counts as helping if it beats those controls.

    05Ranking 3

    Ranking 3: generator apps on their own jobs

    The apps are driven unmodified, with every model call answered by the same model (Opus 5.5) through a relay. "Raw" is the app's output before any agent or fix step.

    06Difficulty

    Baseline difficulty by task

    Click a task to browse every graded output, fully rendered: live apps for G, G2 and I, documents for J, diffs for code tasks.

    A dash (–) means not yet graded. Tasks where Haiku sits below about 95 are where frameworks can show quality lift.

    07Progress

    What has been done

    Eye-test any output:

    ✓ Done
    • Inspected 13 candidates (skill packs, prompt collections, a proxy, generator apps and Meterless), recording their structure, activation paths and red flags.
    • Built 11 tasks with hidden graders, each adversarially audited. Hand-written mediocre solutions score 66–87, which shows the graders discriminate.
    • Pre-registered the arms, controls (minimal, placebo, second review), statistics and composite score, plus four addenda.
    • Ran the Opus 5.5 matrix (A–H complete for every framework), the Haiku 4.5 replication, the relay-served app pipelines (GameMaker, ReMint, Meterless app) and Meterless as a proxy/context layer.
    • Native evidence: headroom replay (cache mode −2.9% tokens; token mode +15.9% cost) and the Meterless engine conformance tests.
    • Integrity work:
      • the attempt audit (one canonical attempt per run);
      • the continued-run detector;
      • exclusion of the Haiku launch confounded by the relayed "status" message;
      • quarantine of infra failures, which are never scored.

    ● Still running or pending

    • Opus: the remaining framework runs on I/J, A, E, F and G2, plus Meterless proxy (batch y01, in progress).
    • Haiku: the full re-run of the excluded launch (a 13-run pilot is in progress, one per arm).
    • Post-steps on Haiku, ceiling-rule top-ups, a quiet-machine regrade of the timing checks, the blind Sonnet judge, and the final report (verdicts, the 14 questions, recommended stack).
    08Read with care

    Caveats