# Nightshift M1-M4 edition: live benchmark, 2026-10-14/15

The shipped `Nightshift M1-M4.app` (v0.90.0) was run live on this Mac across 17 launches plus 8 targeted memory
experiments. All 8 on-device models were measured for speed, memory and capability, along with multi-model
panels, the memory budget, the OpenAI-compatible endpoint, engine jobs and the cloud relay. Seven problems
were found. **All seven are fixed, rebuilt and re-verified live**, and the full test suite passes.

## Summary

| | Result |
| --- | --- |
| Build and tests | 3,281 of 3,285 JS tests pass, 0 fail, 4 skipped (at the start: 1 failing). Rust shell 6/6, macOS-edition tests 14/14, `bundle OK` on all 30 bundle checks, live boot self-test OK |
| Cold boot | Cockpit mounted in **~290 ms** (median of 3 dedicated boot launches); engine ready in ~200 ms; WebLLM worker ready in ~45 ms; 0 page errors |
| Fastest model | **Llama 3.2 1B**: 55–109 tok/s decode (median 90), ~340 tok/s prefill, 0.15–0.19 s to first token |
| Most capable | **Llama 3.2 3B**: 32 of 34 suite prompts correct, at 23–33 tok/s. **Gemma 2 2B** is close behind: 31 of 34 |
| Best code model | **Qwen 2.5 Coder 3B**: 6 of 6 code tasks pass their unit tests |
| Warm model load | 12–23 s for every model, regardless of size. About 11 s of that is spent before any weights are read |
| First download | 8 models, 11.5 GB, took 1 h 52 min on this ~1.2–1.7 MB/s connection, with no failures |
| Stability | **Before:** WebKit killed the engine 19 times in 10 minutes, and the main benchmark could not finish. **After:** every launch on the final build finished with 0 restarts, including the full main run with the window *closed* |
| Privacy | **Before:** a request naming an on-device model sent one step to a cloud model. **After:** every device-model and `meterless/local` request was `local-only` |
| Cloud relay | **Before:** refused (`class_unknown`). **After:** `"NIGHTSHIFT OK"` in 2.7–3.8 s, with no human check needed |

## Problems found, and the fixes

| # | Problem | Fix | Verified live |
| --- | --- | --- | --- |
| 1 | **WebKit killed the whole engine whenever the window was behind another app or closed.** WebKit kills a page over **4 GB when it's "inactive"** (8 GB when active, on Macs with ≤16 GB). GPU memory for the models is charged to that page, and one resident model holds it at 3–5 GB. Every running job was lost: 19 kills in 10 minutes | Occlusion detection off, so a window behind other apps counts as visible. Closing the window **parks** it (transparent, click-through, app steps back) instead of hiding it. The Dock icon, Cmd-Tab or the quit dialog bring it back | `Websam state: active` from launch; 0 restarts with the window covered or closed |
| 2 | **The memory budget allowed model sets that WebKit then killed, even while active.** The largest 16 GB panel it accepted (5.8 GB declared) reached 14.3 GB and was killed twice | **Panel cap.** One model keeps the full budget; two or more must fit 3.0 GB together, at most 2 at once (16 GB). Tiers above 16 GB were scaled to WebKit's 16 GB line (not measured) | Panels trim to 2 models (or 1 under pressure). The two-model peak was 6.7 GB |
| 3 | **Cloud relay models couldn't be named directly.** `meterless-proxy/llama-3.1-8b` was read as the `meterless-<mode>` shorthand: refused, or silently turned into `auto` | The shorthand applies only to ids with no `/` (`js/engine/modes.js`), plus a regression test | Relay call returns `"NIGHTSHIFT OK"` |
| 4 | **One test failed only on macOS**, which spells its temp folder two ways (`/var` and `/private/var`) | The test uses the resolved path | Passes |
| 5 | **best_of_n wrapped one-answer questions in creative briefs**, so small models refused or went off-topic. With n > 1 and no setting, R-ORC-6 raises diversity to `low`, which included "CREATIVE BRIEF … do not produce a generic interpretation" | The automatic raise now uses the temperature ladder **without** creative briefs, so R-ORC-6's goal (no near-identical samples) still holds. An explicit `judge_mode: creative` or `axes.briefs` still gets briefs | "Prime between 10 and 20": refusals and off-topic essays before; **11 / 11 / 11** in 1.0 s after. Muffin problem: three worked solutions |
| 6 | **A request addressed to the device could reach the cloud.** Under AS-31/35 every app is bound to a mode on first contact, and the bind overrides the requested model and mode. A JSON-mode request for `webllm/Llama-3.2-1B` ran a step on `meterless-proxy/qwen3-30b-a3b` (`privacy: edge-mixed`). The README promised the opposite ("routing pins to that model") | A request that names an on-device model, or asks for `meterless/local`, sets a **`local-only` privacy floor**: AS-31's own step (1), which can only tighten. The bind still picks the route. Recorded in the PRD as **AS-35a**; the README now describes binding as it actually works | All three device or local requests `local-only`, on device models only |
| 7 | **Loading a model briefly swelled the page past WebKit's line.** Each weight shard's `ArrayBuffer` stayed charged to the page until GC | A single guarded patch in vendored WebLLM: release each shard (`ArrayBuffer.transfer(0)`) once it's on the GPU. Hash pinned in `macos/pins.env` | See the A/B below: time above the line **≈0.85 s → ≈0.05 s per load** |

### Load-memory A/B (fix 7)

The page's physical footprint was sampled every 20 ms from the kernel (`harness/fpsample.swift`; the same figure
WebKit's kill check reads). The test loaded six models in sequence, two runs per loader:

| Loader | Loads | Mean peak | Worst peak | Time above WebKit's 8 GB line |
| --- | --- | --- | --- | --- |
| Original WebLLM 0.2.84 | 15 | 6.4–6.5 GB | 10.1–10.8 GB | **12.7 s** (≈0.85 s per load) |
| Patched (shard released) | 12 | 5.3–6.0 GB | 7.2–9.0 GB | **0.56 s** (≈0.05 s per load) |

Two other ideas were tested along the way and **reverted** because they measured no effect: removing the
`slice()` copies in the loader, and making GPU uploads zero-copy. Per-load detail is in `raw/98-ab-perload.txt`.

**What remains:** WebKit checks memory about every 30 s, so a kill now needs that check to land in about 0.05 s
of over-the-line time per load, *and* WebKit's own GC-and-shrink pass to fail. That pass recovered 2–5 GB every
time it ran in this study. In the final launches WebKit never killed the page.

## Test machine and method

- **Hardware:** Apple **M5**, 16 GB, 4 performance + 6 efficiency cores, macOS 26.4. This is newer than the M1-M4
  Macs the edition targets. The app's 8 GB and 24 GB tiers were simulated (`NS_MAC_SIMULATE_MEM_GB`); WebKit's
  limits always use the Mac's real RAM.
- **How:** the real app binary, its bundled web content plus one injected harness script, on the app's real
  address, so the model cache, WebLLM worker, budget and engine were exactly what a user gets (`harness/README.md`).
  System memory was sampled every 5 s, page memory every 1 s and every 20 ms, and WebKit's own decisions were read
  from the macOS unified log.
- **Window state:** the app ran behind other apps throughout. The final main runs had the window *closed* from
  30 s in.
- **Grading:** 34 prompts at temperature 0, graded offline (`harness/grade.mjs`). Code answers were run against
  unit tests in Node. Answers were identical before and after the fixes.
- **Variance:** this Mac was also running VS Code, Chrome, Slack and other agents, and was often under `warn`
  memory pressure. Decode speed across 15 launches: Llama 3.2 1B 55–109 tok/s (median 90), SmolLM2 35–74,
  Qwen 1.5B 35–62, Gemma 2B 24–48, the 3B models and Phi 21–33. The appendix tables are from the last run.

## Results

### Device models (final build, 16 GB tier)

| Model | Decode tok/s (median, range) | Prefill tok/s | First token | Warm load | Page memory while resident | Suite (34) | Code (6) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Llama 3.2 1B | **90** (55–109) | 337–345 | 0.15–0.19 s | 13.6–14.4 s | 3.0 GB | 23 (68%) | 3 |
| Qwen 2.5 1.5B | 44 (35–62) | 200–240 | 0.21–0.23 s | 14.6–15.8 s | 3.0 GB | 27 (79%) | 5 |
| SmolLM2 1.7B | 50 (35–74) | 192–215 | 0.29–0.30 s | 13.7–20.2 s | 3.3 GB | 20 (59%) | 4 |
| Gemma 2 2B | 30 (24–48) | 143–150 | 0.35–0.39 s | 17–23 s | 4.5 GB | 31 (91%) | 5 |
| Llama 3.2 3B | 27 (23–33) | 112–121 | 0.45–0.49 s | 14.1–14.8 s | 4.2 GB | **32 (94%)** | 5 |
| Qwen 2.5 3B | 25 (23–25) | 116–122 | 0.47–0.48 s | 15.1–17.2 s | 3.8 GB | 28 (82%) | 4 |
| Qwen 2.5 Coder 3B | 25 (23–25) | 115–123 | 0.48 s | 15.3–17.1 s | 3.9 GB | 29 (85%) | **6** |
| Phi 3.5 Mini | 24 (21–31) | 91–100 | 0.63–0.70 s | 13.5–18.3 s | 5.0 GB | 27 (79%) | 3 |

First-token time with a ~1,460-token prompt: 4.2 s (Llama 1B) to 17.8 s (Phi 3.5). On this hardware the 2–3B
models are the sweet spot. Llama 3.2 3B and Gemma 2 2B are nearly equal in quality, and both answered every
arithmetic and word problem.

### Panels and memory admission

- **16 GB, normal pressure:** asking for 3 small models keeps 2 (Llama 1B + Qwen 1.5B, loaded in 28 s). Running
  both in parallel gave 54.9 tok/s combined vs 47.3 one after the other (+16%). Under `warn` pressure (the last
  run) the same request kept one model, as designed. Phi + Llama 3B + Qwen 3B keeps Phi alone, which still
  answers. All 8 models load on their own.
- **8 GB (simulated):** the 7 models up to 2.5 GB load. Phi 3.5 is refused in **0.06 s** with
  `model_too_large_for_this_mac` before any download, and isn't recommended. Panels shrink to one model.
- **24 GB (simulated):** the planner allows up to 3 models within 6 GB, and both single-model loads tested
  succeeded.

### OpenAI-compatible endpoint (inside the app, `127.0.0.1:8088`)

`/v1/models` (14 entries, 8 device) and `/healthz` answer in under 1 ms. Streaming: first chunk after 5–7 ms.
`meterless/local`: 0.19 s. A request naming a specific device model is still routed by the app's bound mode
(AS-35). It can take 15–40 s while that mode loads a model of its choosing, but it now always stays on the
device.

### Engine jobs on device lanes only (final build)

| Job | Result |
| --- | --- |
| best_of_n, n=3, default settings | Prime question: `11 / 11 / 11` in 1.0 s. Muffin problem: three worked solutions in 12 s |
| best_of_n, n=3, briefs explicitly off | Same answers; the default is now equivalent |
| batch of 3 | Tokyo / Rome / Toronto in 1.0 s. The 1B model got Canada's capital wrong; a model error, not a routing one |
| deliberate | On 1–1.5B lanes, hits the 24-step limit and ends `errored` (74–100 s). One three-lane run finished in 7 s. This is model capability, so it was left as is |

## What changed in the code

| File | Change |
| --- | --- |
| `macos/src-tauri/src/webkit.rs` | `disable_occlusion_detection` (off, then hide and show once so WebKit re-evaluates) and `set_parked` |
| `macos/src-tauri/src/main.rs` | Close parks the window; the Dock icon and focus unpark it; test-only `NS_MAC_TEST_PARK_AFTER_S` |
| `macos/src-tauri/src/quit.rs` | The quit-with-jobs dialog unparks the window first |
| `macos/layer/mac-layer.js` | Panel cap (`panelMb`); 16 GB tier 2 models / 3.0 GB; tiers above 16 GB matched to WebKit's 16 GB limit |
| `macos/pins.env` | WebLLM hash for the patched file, with the reason |
| `vendor/webllm/web-llm.mjs` | One `NIGHTSHIFT PATCH`: release each weight shard after upload |
| `js/engine/modes.js` | `meterless-proxy/<model>` is a relay model id, not a mode |
| `js/engine/jobs.js` | The automatic diversity raise uses no creative briefs |
| `js/engine/proxy.js`, `js/web/web-daemon.js` | `privacyFloorFromRequest`: a device-model or `local` request is `local-only` |
| `tests/…` | `modes-registry` (relay ids), `jobs-diversity` (raise without briefs), `app-stream` (privacy floor), `macos-m1m4` (panel cap), `web-share-export` (macOS temp path) |
| `README.md`, `macos/README.md`, `zDOCS/Fully Implemented/NS_APP_STREAM_JOBS_PRD.md` | How binding and pinning really work; the tier table; closed-window behaviour; the AS-35a amendment |
| `dist-web/` | Regenerated by `tools/build-web.mjs` so the web edition matches the source |

The app and DMG were rebuilt by `macos/build-mac.sh` into `macos/dist/` and `~/nightshift-m1m4-build/dist-mac-m1m4/`.
DMG sha256 `f28ecf7b…79ca`; the original was `35ad26fc…085f`. It is still an ad-hoc-signed, un-notarized
development build. The folder isn't a git repository, so nothing was committed.

**Side effects on this Mac:** the app's storage now holds all 8 models (11.5 GB). Benchmark jobs appear in the
app's run history (`origin.via = "benchmark"`), and the lane setup was restored. The in-app endpoint may have
recorded a mode for the `default` app (AS-34). `~/nightshift-m1m4-build/web-bench` (25 MB) and
`~/nightshift-m1m4-build/fpsample` are harness leftovers and can be deleted.

## Files in this folder

| Path | What |
| --- | --- |
| `BENCHMARK_RESULTS.md` | This report |
| `results.json` | Every graded answer, with its text, plus per-model figures |
| `report-tables.md` | The generated tables (also appended below) |
| `harness/` | `bench.js` (runs inside the app), `run.sh`, `grade.mjs`, `report.mjs`, `loadfp.py` / `loadfps.py` / `loadmem.py`, `fpsample.swift`, `configs/` |
| `raw/` | One log per launch, plus system-memory, page-memory (1 s and 20 ms) and RSS samples; offline test, build and verify logs; the original and patched WebLLM files. `superseded/` holds the earlier main runs. The final main run is `03-main-16gb` (models, panels) + `03-main-16gb-rest` (admission, API, jobs, cloud), split because the session ended mid-run |

To rerun: `bash harness/run.sh <label> harness/configs/<config>.json [simulated GB]`, then `node harness/report.mjs`.

---

## Appendix: generated tables (last run per section)

### Machine and app at launch

| Fact | Value |
| --- | --- |
| Chip | Apple M5 (m5+), 4P + 6E cores |
| Memory | 16 GB → tier **16 GB**: device budget 6000 MB, up to 2 models |
| macOS | 26.4 |
| WebGPU adapter | apple/apple; shader-f16 yes; maxBuffer 2048 MB; maxStorageBinding 2048 MB; workgroup storage 32768 B |
| WebLLM | worker ready 0.063s; 163 prebuilt models in the pinned 0.2.84 build; 8 in the app's device catalog |
| Engine registry | 66 models (8 device, 50 cloud relay); job kinds: deliberate, best_of_n, batch, bulk, app_stream |
| Storage | quota 41.2 GB; usage 11528 MB at launch |

### Cold boot (page timings, per launch)

| Launch | DOMContentLoaded | load event | engine ready | shell mounted | WebLLM worker ready | scripts | page errors |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 90-stress-bg-before-fix | 228 ms | 284 ms | 290 ms | 290 ms | 46 ms | 175 | 0 |
| 90-stress-bg-before-fix | 221 ms | 275 ms | 150 ms | 280 ms | 67 ms | 175 | 0 |
| 90-stress-bg-before-fix | 256 ms | 303 ms | 177 ms | 309 ms | 76 ms | 175 | 0 |
| 90-stress-bg-before-fix | 258 ms | 317 ms | 181 ms | 323 ms | 60 ms | 175 | 0 |
| 90-stress-bg-before-fix | 243 ms | 295 ms | 164 ms | 301 ms | 76 ms | 175 | 0 |
| 90-stress-bg-before-fix | 257 ms | 312 ms | 181 ms | 319 ms | 54 ms | 175 | 0 |
| 90-stress-bg-before-fix | 232 ms | 290 ms | 158 ms | 296 ms | 57 ms | 175 | 0 |
| 90-stress-bg-before-fix | 249 ms | 303 ms | 180 ms | 308 ms | 47 ms | 175 | 0 |
| 90-stress-bg-before-fix | 235 ms | 288 ms | 161 ms | 293 ms | 47 ms | 175 | 0 |
| 90-stress-bg-before-fix | 300 ms | 352 ms | 161 ms | 356 ms | 76 ms | 175 | 0 |
| 90-stress-bg-before-fix | 221 ms | 271 ms | 155 ms | 278 ms | 74 ms | 175 | 0 |
| 90-stress-bg-before-fix | 227 ms | 279 ms | 159 ms | 284 ms | 76 ms | 175 | 0 |
| 90-stress-bg-before-fix | 248 ms | 303 ms | 174 ms | 309 ms | 64 ms | 175 | 0 |
| 90-stress-bg-before-fix | 248 ms | 300 ms | 176 ms | 306 ms | 53 ms | 175 | 0 |
| 90-stress-bg-before-fix | 231 ms | 284 ms | 159 ms | 289 ms | 75 ms | 175 | 0 |
| 90-stress-bg-before-fix | 228 ms | 283 ms | 157 ms | 288 ms | 76 ms | 175 | 0 |
| 90-stress-bg-before-fix | 413 ms | 468 ms | 351 ms | 475 ms | 49 ms | 175 | 0 |
| 90-stress-bg-before-fix | 267 ms | 323 ms | 186 ms | 329 ms | 76 ms | 175 | 0 |
| 90-stress-bg-before-fix | 224 ms | 269 ms | 156 ms | 275 ms | 75 ms | 175 | 0 |
| 90-stress-bg-before-fix | 239 ms | 296 ms | 163 ms | 302 ms | 58 ms | 175 | 0 |
| 03-main-16gb-before-fix | 288 ms | 299 ms | 221 ms | 305 ms | 45 ms | 175 | 0 |
| 03-main-16gb-before-fix | 267 ms | 322 ms | 192 ms | 328 ms | 73 ms | 175 | 0 |
| 03-main-16gb-before-fix | 230 ms | 288 ms | 159 ms | 294 ms | 61 ms | 175 | 0 |
| 03-main-16gb-before-fix | 225 ms | 278 ms | 155 ms | 284 ms | 75 ms | 175 | 0 |
| 03-main-16gb-before-fix | 233 ms | 288 ms | 157 ms | 294 ms | 76 ms | 175 | 0 |
| 03-main-16gb-before-fix | 250 ms | 296 ms | 172 ms | 301 ms | 75 ms | 175 | 0 |
| 03-main-16gb-before-fix | 227 ms | 281 ms | 157 ms | 286 ms | 72 ms | 175 | 0 |
| 03-main-16gb-before-fix | 224 ms | 278 ms | 156 ms | 284 ms | 76 ms | 175 | 0 |
| 91-stress-bg-fixed | 248 ms | 302 ms | 168 ms | 308 ms | 47 ms | 175 | 0 |
| 92-park | 269 ms | 314 ms | 198 ms | 319 ms | 47 ms | 175 | 0 |
| 93-phi | 269 ms | 318 ms | 199 ms | 323 ms | 42 ms | 175 | 0 |
| 94-api-pin | 273 ms | 317 ms | 204 ms | 322 ms | 41 ms | 175 | 0 |
| 95-panel-max | 240 ms | 294 ms | 172 ms | 300 ms | 48 ms | 175 | 0 |
| 95-panel-max | 255 ms | 303 ms | 175 ms | 310 ms | 51 ms | 175 | 0 |
| 95-panel-max | 444 ms | 492 ms | 364 ms | 498 ms | 55 ms | 175 | 0 |
| 96-panel-duo | 232 ms | 283 ms | 161 ms | 289 ms | 40 ms | 175 | 0 |
| 01-validate | 235 ms | 245 ms | 248 ms | 248 ms | 39 ms | 175 | 0 |
| 02-prefetch | 296 ms | 312 ms | 228 ms | 317 ms | 54 ms | 175 | 0 |
| 04-tier-8gb | 234 ms | 281 ms | 287 ms | 287 ms | 46 ms | 175 | 0 |
| 05-tier-24gb | 229 ms | 273 ms | 279 ms | 279 ms | 48 ms | 175 | 0 |
| 06-boot-1 | 277 ms | 334 ms | 203 ms | 340 ms | 43 ms | 175 | 0 |
| 06-boot-2 | 270 ms | 281 ms | 200 ms | 287 ms | 40 ms | 175 | 0 |
| 06-boot-3 | 268 ms | 279 ms | 198 ms | 287 ms | 46 ms | 175 | 0 |
| 03-main-16gb | 321 ms | 373 ms | 230 ms | 377 ms | 39 ms | 175 | 0 |
| 03-main-16gb-rest | 884 ms | 938 ms | 792 ms | 942 ms | 63 ms | 175 | 0 |

Median time to a mounted cockpit: **301 ms** across 45 launches.

### First download (cold, network-bound)

| Model | Was cached | Download + load | Result | Origin storage after |
| --- | --- | --- | --- | --- |
| Llama 3.2 1B | no | 486.7 s | ok | 710 MB |
| SmolLM2 1.7B | no | 562.3 s | ok | 1682 MB |
| Qwen 2.5 1.5B | no | 507.1 s | ok | 2563 MB |
| Gemma 2 2B | no | 868.9 s | ok | 4057 MB |
| Llama 3.2 3B | no | 1051.5 s | ok | 5879 MB |
| Qwen 2.5 3B | no | 1011.1 s | ok | 7628 MB |
| Qwen 2.5 Coder 3B | no | 1240.2 s | ok | 9371 MB |
| Phi 3.5 Mini | no | 1460.3 s | ok | 11528 MB |

### Speed per device model (WebGPU, in-app)

| Model | VRAM (declared) | Warm load | First call | Decode tok/s | Prefill tok/s | TTFT, short | TTFT, ~1458-token prompt | Free mem after load |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Llama 3.2 1B | 0.88 GB | 13.6 s | 0.40 s | **55.5** | 344.9 | 0.19 s | 4.23 s | 47% (normal) |
| Llama 3.2 3B | 2.26 GB | 14.8 s | 0.66 s | **24.3** | 112.3 | 0.45 s | 12.98 s | 49% (normal) |
| Qwen 2.5 1.5B | 1.63 GB | 14.6 s | 0.36 s | **35.2** | 200.6 | 0.21 s | 7.42 s | 51% (normal) |
| Qwen 2.5 3B | 2.50 GB | 17.2 s | 0.69 s | **22.9** | 122.2 | 0.47 s | 12.17 s | 51% (normal) |
| Qwen 2.5 Coder 3B | 2.50 GB | 17.1 s | 0.66 s | **22.8** | 115.1 | 0.48 s | 12.94 s | 50% (normal) |
| Gemma 2 2B | 1.90 GB | 23.3 s | 0.67 s | **24.0** | 143.2 | 0.35 s | 10.41 s | 47% (normal) |
| Phi 3.5 Mini | 3.67 GB | 18.3 s | 0.80 s | **20.9** | 91.4 | 0.70 s | 17.77 s | 41% (normal) |
| SmolLM2 1.7B | 1.77 GB | 20.2 s | 0.52 s | **36.6** | 192.4 | 0.29 s | 7.93 s | 50% (normal) |

### Capability suite (34 auto-graded prompts, temperature 0)

`pass` = correct answer; `strict` = correct and in exactly the requested format. Code answers were executed against unit tests in Node.

| Model | Overall pass | Strict | Math | Reasoning | Knowledge | Instruction | Code | Suite wall time |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Llama 3.2 3B | **32/34** (94%) | 32/34 | 8/8 | 5/6 | 6/6 | 8/8 | 5/6 | 56.4 s |
| Gemma 2 2B | **31/34** (91%) | 29/34 | 8/8 | 5/6 | 6/6 | 7/8 | 5/6 | 93.1 s |
| Qwen 2.5 Coder 3B | **29/34** (85%) | 25/34 | 6/8 | 4/6 | 6/6 | 7/8 | 6/6 | 84.2 s |
| Qwen 2.5 3B | **28/34** (82%) | 28/34 | 6/8 | 4/6 | 6/6 | 8/8 | 4/6 | 85.4 s |
| Qwen 2.5 1.5B | **27/34** (79%) | 25/34 | 4/8 | 5/6 | 6/6 | 7/8 | 5/6 | 51.6 s |
| Phi 3.5 Mini | **27/34** (79%) | 23/34 | 8/8 | 4/6 | 6/6 | 6/8 | 3/6 | 136.3 s |
| Llama 3.2 1B | **23/34** (68%) | 20/34 | 5/8 | 1/6 | 6/6 | 8/8 | 3/6 | 23.9 s |
| SmolLM2 1.7B | **20/34** (59%) | 19/34 | 5/8 | 2/6 | 3/6 | 6/8 | 4/6 | 46.7 s |

<details><summary>Per-item results (✓ pass · ✓✓ strict · ✗ fail)</summary>

| Item | Llama 3.2 1B | Llama 3.2 3B | Qwen 2.5 1.5B | Qwen 2.5 3B | Qwen 2.5 Coder 3B | Gemma 2 2B | Phi 3.5 Mini | SmolLM2 1.7B |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| math_mul | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ |
| math_add | ✗ | ✓✓ | ✗ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✗ |
| math_pct | ✗ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✗ |
| math_pow | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓ |
| gsm_muffins | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓ | ✓✓ | ✓✓ | ✗ |
| gsm_apples | ✓ | ✓✓ | ✗ | ✗ | ✓ | ✓✓ | ✓✓ | ✓✓ |
| gsm_train | ✓✓ | ✓✓ | ✗ | ✗ | ✗ | ✓✓ | ✓✓ | ✓✓ |
| gsm_price | ✗ | ✓✓ | ✗ | ✓✓ | ✗ | ✓✓ | ✓✓ | ✓✓ |
| logic_syllogism | ✗ | ✗ | ✓✓ | ✗ | ✓✓ | ✓✓ | ✓ | ✓✓ |
| logic_feathers | ✗ | ✓✓ | ✗ | ✗ | ✗ | ✓✓ | ✓ | ✓✓ |
| logic_daughters | ✗ | ✓✓ | ✓✓ | ✓✓ | ✗ | ✓✓ | ✗ | ✗ |
| logic_order | ✗ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✗ | ✓✓ | ✗ |
| logic_sequence | ✗ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✗ | ✗ |
| logic_apples | ✓✓ | ✓✓ | ✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✗ |
| kn_canberra | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✗ |
| kn_gold | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✗ |
| kn_austen | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ |
| kn_jupiter | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ |
| kn_ww2 | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ |
| kn_boil | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✗ |
| if_banana | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ |
| if_colors | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ |
| if_five_words | ✓✓ | ✓✓ | ✗ | ✓✓ | ✗ | ✗ | ✗ | ✗ |
| if_json | ✓ | ✓✓ | ✓✓ | ✓✓ | ✓ | ✓ | ✓ | ✓✓ |
| if_extract | ✓ | ✓✓ | ✓ | ✓✓ | ✓ | ✓ | ✓ | ✓✓ |
| if_sentiment | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ |
| if_translate | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ |
| if_summary | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✗ | ✗ |
| code_palindrome | ✗ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✗ | ✓✓ |
| code_fizzbuzz | ✗ | ✗ | ✗ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✗ |
| code_sumevens | ✓✓ | ✓✓ | ✓✓ | ✗ | ✓✓ | ✓✓ | ✗ | ✓✓ |
| code_fib | ✗ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✗ | ✓✓ | ✓✓ |
| code_vowels | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ |
| code_dedupe | ✓✓ | ✓✓ | ✓✓ | ✗ | ✓✓ | ✓✓ | ✗ | ✗ |

</details>

### Multi-model panel (models resident at once)

| Measure | Value |
| --- | --- |
| Requested | Llama 3.2 1B + Qwen 2.5 1.5B + SmolLM2 1.7B |
| Resident after load | Llama 3.2 1B |
| Panel load (warm, from cache) | 23.5 s |
| Trim | kept 1, dropped Qwen 2.5 1.5B, SmolLM2 1.7B (pressure warn, budget 3600 MB) |
| Sequential, 3×128 tokens | 2.9 s → 43.8 tok/s aggregate |
| Parallel, 3×128 tokens | 2.9 s → 44.4 tok/s aggregate |
| Free memory with panel resident | 56% (normal) |

### Over-budget panel request

| Measure | Value |
| --- | --- |
| Requested | Phi 3.5 Mini + Llama 3.2 3B + Qwen 2.5 3B |
| Planner | keep Phi 3.5 Mini (3672 of 6000 MB) · drop Llama 3.2 3B, Qwen 2.5 3B |
| Resident after load | Phi 3.5 Mini |
| Trim recorded (RT.mac.lastTrim) | yes: dropped Llama 3.2 3B, Qwen 2.5 3B |
| Load | 21.1 s |
| Still answers | "Panel status is confirmed as operational" |

### Admission control (device-model budget, live)

**04-tier-8gb**: tier 8 GB, budget 2600 MB, max 1 model(s). Recommended downloads: Llama 3.2 1B, Llama 3.2 3B, Qwen 2.5 1.5B, Qwen 2.5 3B, Qwen 2.5 Coder 3B, Gemma 2 2B, SmolLM2 1.7B

| Model | Declared VRAM | Fits tier | Live outcome | Code | Time to outcome | Catalog flag |
| --- | --- | --- | --- | --- | --- | --- |
| Llama 3.2 1B | 879.04 MB | yes | loaded | — | 13.00 s | fits |
| Qwen 2.5 1.5B | 1629.75 MB | yes | loaded | — | 15.18 s | fits |
| SmolLM2 1.7B | 1774.19 MB | yes | loaded | — | 14.39 s | fits |
| Gemma 2 2B | 1895.3 MB | yes | loaded | — | 14.40 s | fits |
| Llama 3.2 3B | 2263.69 MB | yes | loaded | — | 14.05 s | fits |
| Qwen 2.5 3B | 2504.76 MB | yes | loaded | — | 13.39 s | fits |
| Qwen 2.5 Coder 3B | 2504.76 MB | yes | loaded | — | 13.07 s | fits |
| Phi 3.5 Mini | 3672.07 MB | no | refused | model_too_large_for_this_mac | 0.06 s | too large |

**05-tier-24gb**: tier 24 GB, budget 9000 MB, max 3 model(s). Recommended downloads: Llama 3.2 1B, Llama 3.2 3B, Qwen 2.5 1.5B, Qwen 2.5 3B, Qwen 2.5 Coder 3B, Gemma 2 2B, Phi 3.5 Mini, SmolLM2 1.7B

| Model | Declared VRAM | Fits tier | Live outcome | Code | Time to outcome | Catalog flag |
| --- | --- | --- | --- | --- | --- | --- |
| Llama 3.2 1B | 879.04 MB | yes | loaded | — | 11.19 s | fits |
| Phi 3.5 Mini | 3672.07 MB | yes | loaded | — | 14.35 s | fits |

**03-main-16gb-rest**: tier 16 GB, budget 6000 MB, max 2 model(s). Recommended downloads: Llama 3.2 1B, Llama 3.2 3B, Qwen 2.5 1.5B, Qwen 2.5 3B, Qwen 2.5 Coder 3B, Gemma 2 2B, Phi 3.5 Mini, SmolLM2 1.7B

| Model | Declared VRAM | Fits tier | Live outcome | Code | Time to outcome | Catalog flag |
| --- | --- | --- | --- | --- | --- | --- |
| Llama 3.2 1B | 879.04 MB | yes | loaded | — | 16.70 s | fits |
| Qwen 2.5 1.5B | 1629.75 MB | yes | loaded | — | 18.44 s | fits |
| SmolLM2 1.7B | 1774.19 MB | yes | loaded | — | 21.86 s | fits |
| Gemma 2 2B | 1895.3 MB | yes | loaded | — | 19.90 s | fits |
| Llama 3.2 3B | 2263.69 MB | yes | loaded | — | 18.05 s | fits |
| Qwen 2.5 3B | 2504.76 MB | yes | loaded | — | 19.90 s | fits |
| Qwen 2.5 Coder 3B | 2504.76 MB | yes | loaded | — | 18.84 s | fits |
| Phi 3.5 Mini | 3672.07 MB | yes | loaded | — | 19.89 s | fits |

### Budget planner matrix (pure function, all tiers × memory pressure)

Cells: models kept / requested · MB used of budget.

| Mac | Pressure | single_1b | single_3b | single_phi | trio_small | duo_3b | phi_plus_small | all_eight |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 8 GB | normal | 1/1 · 879/2600 | 1/1 · 2264/2600 | 0/1 · 0/2600 | 1/3 · 879/2600 | 1/2 · 2264/2600 | 1/2 · 879/2600 | 1/8 · 879/2600 |
| 8 GB | warn | 1/1 · 879/1560 | 0/1 · 0/1560 | 0/1 · 0/1560 | 1/3 · 879/1560 | 0/2 · 0/1560 | 1/2 · 879/1560 | 1/8 · 879/1560 |
| 8 GB | critical | 0/1 · 0/0 | 0/1 · 0/0 | 0/1 · 0/0 | 0/3 · 0/0 | 0/2 · 0/0 | 0/2 · 0/0 | 0/8 · 0/0 |
| 16 GB | normal | 1/1 · 879/6000 | 1/1 · 2264/6000 | 1/1 · 3672/6000 | 2/3 · 2509/6000 | 1/2 · 2264/6000 | 1/2 · 3672/6000 | 2/8 · 2509/6000 |
| 16 GB | warn | 1/1 · 879/3600 | 1/1 · 2264/3600 | 0/1 · 0/3600 | 1/3 · 879/3600 | 1/2 · 2264/3600 | 1/2 · 879/3600 | 1/8 · 879/3600 |
| 16 GB | critical | 0/1 · 0/0 | 0/1 · 0/0 | 0/1 · 0/0 | 0/3 · 0/0 | 0/2 · 0/0 | 0/2 · 0/0 | 0/8 · 0/0 |
| 24 GB | normal | 1/1 · 879/9000 | 1/1 · 2264/9000 | 1/1 · 3672/9000 | 3/3 · 4283/9000 | 2/2 · 4768/9000 | 2/2 · 4551/9000 | 3/8 · 4772/9000 |
| 24 GB | warn | 1/1 · 879/5400 | 1/1 · 2264/5400 | 1/1 · 3672/5400 | 1/3 · 879/5400 | 1/2 · 2264/5400 | 1/2 · 3672/5400 | 1/8 · 879/5400 |
| 24 GB | critical | 0/1 · 0/0 | 0/1 · 0/0 | 0/1 · 0/0 | 0/3 · 0/0 | 0/2 · 0/0 | 0/2 · 0/0 | 0/8 · 0/0 |
| 32 GB | normal | 1/1 · 879/9000 | 1/1 · 2264/9000 | 1/1 · 3672/9000 | 3/3 · 4283/9000 | 2/2 · 4768/9000 | 2/2 · 4551/9000 | 3/8 · 4772/9000 |
| 32 GB | warn | 1/1 · 879/5400 | 1/1 · 2264/5400 | 1/1 · 3672/5400 | 1/3 · 879/5400 | 1/2 · 2264/5400 | 1/2 · 3672/5400 | 1/8 · 879/5400 |
| 32 GB | critical | 0/1 · 0/0 | 0/1 · 0/0 | 0/1 · 0/0 | 0/3 · 0/0 | 0/2 · 0/0 | 0/2 · 0/0 | 0/8 · 0/0 |

### OpenAI-compatible surface (in-tab daemon, `127.0.0.1:8088`)

| Call | Result |
| --- | --- |
| GET /v1/models | 200, 14 models in 0.00 s (device: 8) |
| GET /healthz | 200 in 0.00 s |
| POST chat (device model, non-stream) | 200 in 38.47 s → "Paris." · routed **Llama-3.2-3B-Instruct-q4f16_1-MLC** via `meterless/auto` (final_synthesis=Llama-3.2-3B-Instruct-q4f16_1-MLC); privacy `local-only` |
| POST chat (response_format json_object) | 200 in 15.60 s → "{   "ok": true,   "n": 3 }" · routed **Llama-3.2-1B-Instruct-q4f16_1-MLC** via `meterless/auto` (tool_call_format=Llama-3.2-1B-Instruct-q4f16_1-MLC); privacy `local-only` |
| POST chat (meterless/local mode) | 200 in 0.18 s → "42" · routed **Llama-3.2-1B-Instruct-q4f16_1-MLC** via `meterless/auto` (final_synthesis=Llama-3.2-1B-Instruct-q4f16_1-MLC); privacy `local-only` |
| POST chat (stream) | 200; first chunk 0.01 s; 4 chunks in 0.90 s → "Here is the count from 1 to 10, separated by spaces:  1 2 3 " |

### Engine jobs on device lanes only

Lanes: llama-lane (meterless-proxy/llama-3.1-8b, race), qwen-lane (meterless-proxy/qwen3-30b-a3b, race), glm-lane (meterless-proxy/glm-5.3-flash, race), mistral-lane (meterless-proxy/mistral-small-24b, race), device-llama-3-2-1b-instruct-q4f16-1-mlc (Llama-3.2-1B-Instruct-q4f16_1-MLC, race), bench-0 (Llama-3.2-1B-Instruct-q4f16_1-MLC, race), bench-1 (Qwen2.5-1.5B-Instruct-q4f16_1-MLC, race); panel settled in 53.2 s.

| Launch | Job | Status | Wall time | Attempts | Winner | Answer(s) | Events |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 01-validate | best_of_n_n3 | completed | 4.0 s | 4 | 3 | `I cannot produce a sentence that reverse` / `I cannot provide a response that include` / `I cannot provide an answer that does not` / `# cluster 0 · "unnamed" · 3 members · 2 ` | 24 |
| 01-validate | deliberate | errored | 93.4 s | 7 | 24 | `COMPLETED: Unit tests should be written ` / `COMPLETED: Unit tests are a crucial part` / `COMPLETED: The team should consider the ` / `COMPLETED: Deciding whether to write uni` / `COMPLETED: The development team should c` / `S: The team should consider writing unit` / `COMPLETED: <1> Unit tests REMAINING: <Un` | 99 |
| 01-validate | batch_3 | completed | 1.0 s | 3 | — | `Tokyo.` / `Rome.` / `Ottawa.` | 14 |
| 03-main-16gb-rest | best_of_n_n3_prime (default diversity) | completed | 1.0 s | 3 | 1 | `11` / `11` / `11` | 22 |
| 03-main-16gb-rest | best_of_n_n3_prime (briefs off) | completed | 1.0 s | 3 | 1 | `11` / `11` / `11` | 19 |
| 03-main-16gb-rest | best_of_n_n3_math (default diversity) | completed | 12.1 s | 4 | 2 | `To find out how many muffins are left, w` / `To find out how many muffins are left, w` / `To determine how many muffins are left a` / `# cluster 0 · "unnamed" · 3 members · 3 ` | 24 |
| 03-main-16gb-rest | best_of_n_n3_math (briefs off) | completed | 10.0 s | 4 | 1 | `To find out how many muffins are left, w` / `To find out how many muffins are left, w` / `To find out how many muffins are left af` / `# cluster 0 · "unnamed" · 3 members · 3 ` | 24 |
| 03-main-16gb-rest | deliberate | errored | 100.5 s | 7 | 24 | `COMPLETED: Implemented the first prototy` / `COMPLETED: Implemented the first prototy` / `The team has decided to prioritize the i` / `COMPLETED: Implemented the first prototy` / `COMPLETED: Implementation of unit tests ` / `` / `COMPLETED: The team needs to define and ` | 99 |
| 03-main-16gb-rest | batch_3 | completed | 1.0 s | 3 | — | `Tokyo.` / `Rome.` / `Toronto.` | 12 |

### Cloud relay lane

| Launch | Relay configured | Result | Time |
| --- | --- | --- | --- |
| 01-validate | yes | **{"message":"\"meterless/proxy/llama-3\" is not a mode. See GET /v1/meterless/modes.","type":"invalid_request_error","code":"class_unknown","param":"model"}** | 0.0 s |
| 03-main-16gb-rest | yes | ok: "NIGHTSHIFT OK" | 3.8 s |

### System memory while each launch ran (sampled every 5 s)

| Launch | Duration | Lowest free | Worst pressure | Peak swap used | Peak WebKit RSS (CPU side) |
| --- | --- | --- | --- | --- | --- |
| 90-stress-bg-before-fix | 10 min | 27% | **warn or worse** | 30590 MB | 8410 MB |
| 03-main-16gb-before-fix | 11 min | 30% | **warn or worse** | 9534 MB | 9245 MB |
| 91-stress-bg-fixed | 2 min | 40% | normal | 3836 MB | 8829 MB |
| 92-park | 1 min | 48% | normal | 3168 MB | 8405 MB |
| 93-phi | 1 min | 43% | normal | 3144 MB | 7441 MB |
| 94-api-pin | 3 min | 40% | **warn or worse** | 3725 MB | 9193 MB |
| 95-panel-max | 3 min | 25% | **warn or worse** | 14671 MB | 7996 MB |
| 96-panel-duo | 2 min | 38% | **warn or worse** | 3707 MB | 9009 MB |
| 01-validate | 10 min | 36% | **warn or worse** | 1355 MB | 3812 MB |
| 02-prefetch | 112 min | 40% | **warn or worse** | 1311 MB | 6785 MB |
| 04-tier-8gb | 3 min | 25% | **warn or worse** | 4190 MB | 8624 MB |
| 05-tier-24gb | 0 min | 44% | normal | 4078 MB | 8366 MB |
| 06-boot-1 | 0 min | 76% | normal | 3966 MB | 495 MB |
| 06-boot-2 | 0 min | 77% | normal | 3926 MB | 505 MB |
| 06-boot-3 | 0 min | 77% | normal | 3902 MB | 477 MB |
| 03-main-16gb | 21 min | 22% | **warn or worse** | 6129 MB | 6793 MB |
| 03-main-16gb-rest | 7 min | 30% | **warn or worse** | 7803 MB | 8726 MB |

### Web-process memory per resident model (what WebKit's kill line counts)

| Launch | Model | Declared VRAM | WebContent footprint, median | Peak | Samples |
| --- | --- | --- | --- | --- | --- |
| 90-stress-bg-before-fix | Gemma 2 2B | 1.90 GB | 0.7 GB | 5.2 GB | 7 |
| 91-stress-bg-fixed | Gemma 2 2B | 1.90 GB | 2.5 GB | 7.9 GB | 14 |
| 91-stress-bg-fixed | Phi 3.5 Mini | 3.67 GB | 1.2 GB | 5.2 GB | 9 |
| 92-park | Gemma 2 2B | 1.90 GB | 1.8 GB | 4.8 GB | 11 |
| 92-park | Llama 3.2 3B | 2.26 GB | 4.4 GB | 4.4 GB | 1 |
| 93-phi | Phi 3.5 Mini | 3.67 GB | 4.9 GB | 4.9 GB | 1 |
| 03-main-16gb | Llama 3.2 1B | 0.88 GB | 2.7 GB | 3.4 GB | 21 |
| 03-main-16gb | Qwen 2.5 1.5B | 1.63 GB | 3.1 GB | 5.1 GB | 31 |
| 03-main-16gb | SmolLM2 1.7B | 1.77 GB | 3.4 GB | 4.7 GB | 35 |
| 03-main-16gb | Gemma 2 2B | 1.90 GB | 4.5 GB | 4.7 GB | 52 |
| 03-main-16gb | Llama 3.2 3B | 2.26 GB | 4.0 GB | 4.8 GB | 43 |
| 03-main-16gb | Qwen 2.5 3B | 2.50 GB | 3.8 GB | 4.0 GB | 54 |
| 03-main-16gb | Qwen 2.5 Coder 3B | 2.50 GB | 3.8 GB | 5.1 GB | 57 |
| 03-main-16gb | Phi 3.5 Mini | 3.67 GB | 5.2 GB | 5.6 GB | 42 |

### Launch stability (each page start after the first = WebKit killed the web process)

| Launch | Page starts | Restarts | Finished | Peak WebContent footprint | Web processes seen |
| --- | --- | --- | --- | --- | --- |
| 90-stress-bg-before-fix | 20 | 19 | **no** | 7.9 GB | 20 |
| 03-main-16gb-before-fix | 8 | 7 | **no** | — | — |
| 91-stress-bg-fixed | 1 | 0 | yes | 7.9 GB | 1 |
| 92-park | 1 | 0 | yes | 7.0 GB | 1 |
| 93-phi | 1 | 0 | yes | 5.0 GB | 1 |
| 94-api-pin | 1 | 0 | yes | 9.0 GB | 1 |
| 95-panel-max | 3 | 2 | yes | 14.3 GB | 3 |
| 96-panel-duo | 1 | 0 | yes | 6.7 GB | 1 |
| 01-validate | 1 | 0 | yes | — | — |
| 02-prefetch | 1 | 0 | yes | — | — |
| 04-tier-8gb | 1 | 0 | yes | 9.1 GB | 1 |
| 05-tier-24gb | 1 | 0 | yes | 5.9 GB | 1 |
| 06-boot-1 | 1 | 0 | yes | 0.8 GB | 1 |
| 06-boot-2 | 1 | 0 | yes | 0.8 GB | 1 |
| 06-boot-3 | 1 | 0 | yes | 0.8 GB | 1 |
| 03-main-16gb | 1 | 0 | **no** | 7.1 GB | 1 |
| 03-main-16gb-rest | 1 | 0 | yes | 9.9 GB | 1 |