Nightshift M1-M4,
measured inside the real app.
The shipped app ran on this Mac for 17 launches: 8 on-device models, a 34-prompt graded suite, memory traced every 20 ms, the OpenAI endpoint, engine jobs and the cloud relay. Seven problems turned up. All seven are fixed and re-checked live.
Speed against smarts
Each dot is a model: its median decode speed across every launch, and how many of the 34 graded prompts it got right. The whisker is the slowest-to-fastest spread on a busy Mac. Click a model to follow it through the rest of the page.
Decode speed vs suite score
Hover for details · click to selectMetric
Model table
Every answer, graded
Solid green means right and in exactly the requested format; outlined means right but loose on format; red means wrong. Code answers were run against unit tests. Click any cell to read the model's actual answer.
The engine kept getting killed
WebKit kills a page that grows past 4 GB while its window is hidden or covered, or 8 GB while it's visible. The models' GPU memory counts against the page, so with the window behind another app, every 2B-plus model crossed the line and WebKit took the engine down with every running job. The shell now keeps the page visible to WebKit, and closing the window parks it instead of hiding it.
Load spikes, before and after the loader patch
Loading a model briefly swells the page. Each weight shard stayed charged to it until garbage collection ran. One patch to WebLLM releases each shard as soon as it's on the GPU. Pick a model to overlay every recorded load of it.
All 27 recorded loads
Two other changes were tried and reverted because they measured no effect.
Try the memory planner
The app decides which models it may hold at once before it loads anything. The old 16 GB plan allowed three models and 6 GB; the biggest panel it accepted reached 14.3 GB and was killed twice. The new plan gives one model the full budget, but two or more must fit 3 GB together. Click models in priority order and compare.
Seven problems
Click a card for the detail.
Endpoint and engine jobs
Requests that name an on-device model now carry a local-only privacy floor; before the fix, one of these sent a step to a cloud model. Multi-attempt jobs no longer wrap one-answer questions in creative briefs, which made the small models refuse.
OpenAI-compatible endpoint
Jobs on device lanes
Seventeen launches
Each dot is one time the page started. Red dots are restarts after WebKit killed the engine. The first three launches ran before the fixes.
Cold boot, three dedicated launches
Measured from navigation start inside the app's web view.