# Benchmark harness: Nightshift M1-M4 edition, live

This harness runs the shipped `Nightshift M1-M4.app` itself, not a browser and not a mock. `run.sh` copies the
bundle's own web root to `~/nightshift-m1m4-build/web-bench`, adds one script (`bench.js`) after
`js/mac/mac-layer.js`, and launches the bundle's binary with `NS_MAC_WEB_ROOT` pointed at that copy. The page is
served on the app's real origin, `http://127.0.0.1:47321`. It uses the same model cache, the same WebLLM worker,
the same device-model budget, and the same in-tab engine and OpenAI surface that a user gets.

`bench.js` reads `bench-config.json`, runs the listed phases, and posts every result to the shell's
`/__ns/diag` route. The shell writes those results to stderr (`raw/<label>.log`). While a launch runs, `run.sh`
samples system memory every 5 s (`raw/<label>-sys.tsv`), and kills the app once the page reports `done`.

| Phase | What it measures |
| --- | --- |
| `env` | Chip, tier, WebGPU adapter (features, limits), WebLLM worker start-up, page boot timings, storage, engine registry |
| `planner` | The pure budget planner (`RT.macPlan`) over every tier × pressure × panel shape |
| `prefetch` | First download plus load of each model (network-bound) |
| `models` | Per model: warm load from cache, first call, decode tok/s (256 tokens, 2 reps), prefill tok/s (~1.5k-token prompt, 2 reps), 34-prompt capability suite at temperature 0 |
| `panel` | Several models resident at once: load, sequential vs parallel throughput, plus an over-budget request that must be trimmed |
| `admission` | Every catalog model against the tier: loads or is refused with `model_too_large_for_this_mac` before any download |
| `api` | The in-tab OpenAI surface: `/v1/models`, `/healthz`, chat (non-stream, JSON mode, `meterless/local`, SSE stream) |
| `jobs` | Real engine jobs (`best_of_n`, `deliberate`, `batch`) on device lanes only. The user's lane store is restored afterwards |
| `cloud` | One relay call (`meterless-proxy/llama-3.1-8b`) through the in-tab daemon |

Rerun one launch:

```sh
bash run.sh <label> configs/<config>.json [simulate_mem_gb] [timeout_min]
node report.mjs            # → ../report-tables.md, ../results.json
```

`grade.mjs` grades the suite offline. Numeric answers use the last number, or the number after "Answer:". Code
answers are executed in a Node `vm` against unit tests. `strict` additionally requires the exact requested format
(and, for code, the exact function name).