Nightshift for Windows: benchmark findings

The short version

Nightshift runs many attempts at a job across several local models, clusters and judges them, and records whether the result is good enough to keep. We ran the Mac campaign's harnesses, ported to Windows, against the installed app on a gaming PC with a 24 GB RTX 3090, in daily use.

On a 24 GB card, the catalog fleets that fit run in seconds. Twelve catalog fleets, three jobs each, three times over: 108 of 108 jobs completed, and 100 of 108 crowned winners were correct (93 %, against 87 % for a random attempt). The judge panel ran on every job. Eight of the twelve fleets keep every lane on the card at once. A median job took 3–33 s; the same fleets took 6–472 s on the 16 GB Mac, so the PC is 1.5–115× faster depending on whether the Mac had to rotate models.

It holds up for hours. A 3-hour back-to-back run completed 553 of 553 jobs (3 318 attempts, 0 errors) at 185, 188 and 180 jobs an hour, with the runtime's control channel answering every check and the lanes' memory flat from start to finish. The Mac's numeric release gates pass 7 of 7, with a median model switch of 3.3 s against a 7 s gate.

Heat and energy, measured. A heat brake at the card's own limit held the card at 76 °C under an 80 °C owner limit, and never engaged in normal work, up to 83 °C. GPU energy: about 0.3–0.4 Wh per resident best-of-4 job, about 1.3 Wh per best-of-6 job with 512-token answers.

What is weak. The judge: shown a correct answer, the 3B judge still crowns another in 19 of 42 jobs, and fleets with no capable judge end with unparseable panels. "Verified" for a generated app means statically valid: 10 of 12 apps were verified and 6 worked. Per-job synthesis needs room for an 8B curator; on a busy desktop most jobs went without it. And a PC's GPU is shared: other programs take VRAM, compute and memory, and Nightshift's plans shrink to match.

Every number names its raw folder. Failures, refusals and timeouts stay in the denominators.

At a glance

1. The machine and how it was measured

The machine. A desktop PC in daily use: AMD Ryzen 9 5900X (12 cores), 64 GB RAM, NVIDIA GeForce RTX 3090 24 GB (driver 560.94, 350 W, NVML slowdown temperature 95 °C), Windows 10 Pro 22H2. A browser, VS Code and several AI desktop apps were open throughout; the owner also ran a Whisper speech server (2 GB of VRAM) and, at times, a game emulator, a local bridge server and test runs. The models were on the internal NVMe drive (C:\NightshiftModels).

How Windows differs from the Mac and the DGX. The same product and code, with Windows behaviour behind cfg(windows). A discrete GPU has its own 24 GB of VRAM beside system RAM, so the planner packs fleets against the VRAM free right now (21.4 GB with the desktop open) and treats RAM only as spill. Commit charge, not swap pressure, is the hard memory limit. Lanes run CUDA on NVIDIA cards, Vulkan elsewhere.

What ran. The Mac campaign's harnesses, ported, against the installed app through its public API and control pipe:

Scoring. "Winner correct" scores the attempt the product crowned, not the best one it had; "candidates correct" scores every attempt. The suites are small and on the easy side, as on the Mac: they measure whether the runtime delivers a fleet's ability, not how capable the models are.

Caveats. One PC, one card: a 12 GB or 8 GB card would rotate most of these fleets. A working desktop: substrate numbers move by a third from run to run. Licence opt-ins were left as the owner had them (the Liquid models are off, and lanes that pin them sit out of the race). Token rates are the product's estimates; energy is the GPU board only.

2. Results

2.1 Every catalog fleet that fits, three times over

12 catalog fleets × 3 jobs × 3 repetitions, best_of_n n = 4 (fleets/w5-r1..r3). The owner's Whisper server held 2 GB of VRAM throughout.

• Every job completed, and the judge panel ran on all 108. The crowned winner was correct 93 % of the time, against 87 % for a random attempt from the same jobs. Selection adds a little on an easy suite, as on the Mac (90 % vs 82 %).
• Eight of the twelve fleets race with every lane on the card at once. The Mac rotated four of those eight, so the gap is widest there (ext-lite 3.3 s against 379 s).
• Four fleets still rotate on 24 GB with the desktop open (testgen-lite, ops-lite, web-bulk-lite, ext-core). They finish in 9–33 s; the model switching inside a rotated job costs a median 21 s.
• Where both machines run concurrently (starter-16, percept-lite, xfer-lite, asr-factory), the PC is 1.5–3× faster: decode speed, not planning, is the difference.
• The 8 wrong winners: asr-factory 3, ext-lite 2, starter-16 1, xfer-lite 1, ops-lite 1. asr-factory seats only a 1B "thinking" router and a 3B reasoning verifier, neither of which can judge, so its panels came back unparseable.
• Every job record carries the fleet that was active when it ran (108 of 108).

2.2 Job modes on the controlled fleet (the grid)

The Mac's modes grid unchanged: 52 cells × 3 repetitions = 156 jobs, deterministic scorers (modes/grid-w5). The fleet is resident for every job on this card (155 of 156 plans concurrent).

• The panel sees the right answer; the judge is the limit. Almost every job lands in one lexical cluster (74 of 84 panel jobs), and the panel shows the judge up to six distinct attempts across lanes. It was shown a correct candidate in 42 of the 45 jobs that had one; the 3B judge then picked another in 19 of those 42.
• Speed is the card, not the mode. Every mode ran in seconds, so deadlines of 45 s and 120 s were never the constraint.
• Weaknesses shared with the Mac: a quality rubric passes an invoice whose line items are wrong; the consistency judge flags some true answers (2 of 3 consistent prompts); more diversity costs tokens without improving the winner.

2.3 Makes, repair, synthesis, lineage and Auto

The Mac's extras steps on the modes fleet (modes/extras-w5), with a real headless Chrome for the app checks.

2.4 Bulk, pause and resume, and a crash mid-chain

A crash mid-chain. A deliberate chain was killed (the engine host process, taskkill /F) after its first chunk landed and before any checkpoint was written. The app restarted the engine host by itself and the chain restarted at chunk 1, the correct behaviour with no checkpoint to resume from, and completed. Forking from a chunk that was never written was refused by name, never guessed.

2.5 Watchers and schedules

A filesystem watcher on the 20-receipt folder (5-minute cadence, debounce of 2 stable observations) beside an interval schedule every 5 minutes, 46 minutes in all (modes/watch-w6):

3 real changes → 3 fires, 0 false fires, 0 duplicates. The interval schedule ran 9 of 9 times, 300–303 s apart; seven of its runs took 3.6–4.3 s, and the two that shared the GPU with another long run took 33 s and 233 s.

2.6 The quiet window and the idle gate

Two extraction jobs submitted at 21:04 local (modes/quietwin-w6), on a box that was busy with a long run:

Both holds did what they say: the quiet job took the first free slot after 22:00, and the idle-gated job waited for an idle box, not merely for the window.

2.7 A 3-hour sustained run

best_of_n n = 6 jobs back to back on five rotating prompts (512-token answers), on ext-lite, with the runtime's status sampled every 30 s and the GPU every second (sustained/, 07:25–10:26Z):

• No failure, stall or slowdown. Every job completed, the third hour ran at the rate of the first, and the status channel answered all 361 samples.
• Memory stayed put. The seven lanes held the same 20.3 GB of commit from the first sample to the last. When system commit crossed 85 % of its limit (other apps growing), the posture went to Reduce once, between two samples, and back to Normal within 15 healthy ticks; the 13 GB curator was refused a start for commit 34 times rather than pushed into a nearly full commit limit.
• Per-job synthesis needs a seat. Each job also queues a synthesis on an 8B curator. With the fleet holding the card and the desktop holding memory, the curator rarely got room: 133 of the 553 jobs got a finished synthesis. On the final build the queue stops at 16 nights waiting on one curator, and later nights record curator_behind (a 1-hour run: the backlog held at 16 for 52 minutes and 200 nights recorded the reason). The jobs' winners never depend on it.

2.8 Restart with queued work

The app was relaunched with the owner's own lanes while 21 queued synthesis rows waited for a curator that was no longer configured, and /health was probed once a second for 5 minutes (boot-b12.jsonl). The API answered 289 of 289 probes after first contact at 9 s: p50 16 ms, p95 104 ms, p99 479 ms, worst 1.5 s. Its startup work ran meanwhile: the 17 orphaned makes failed by name, and 25 archived nights were indexed.

2.9 Release gates and quitting the app

The Mac's numeric release gates (tools/mac/release-gates.mjs --model extract, ext-lite's Llama-3.2-1B lane, gates/gates-b12.json), unchanged:

All 7 pass. Measured straight after a burst of jobs, the gates can fail: the engine host keeps working on the finished jobs (enrichment, consistency checks) for some minutes, and a run started then measured generation at 41.7 tok/s and health under load at 1 474 ms.

Quitting. With nothing running, closing the app window stops the background runtime with it: in two tries the app exited in 2 s, and the runtime logged its shutdown and was gone within the same 2 s, lanes and engine included. When a job is working, the runtime keeps going in the background and the app says so.

2.10 Substrate: what the card does with one model

llama-bench from the app's own llama.cpp build, the lane flags (flash attention, q8_0 KV cache), all layers on the GPU, 3 repetitions, with the starter-16 lanes resident but idle (substrate/).

• Working-desktop numbers: back-to-back runs of the 1B decoded at 233 and 355 tok/s, and of the 8B at 70 and 53. Read the table as magnitudes.
• Against the Mac: the M5 decodes the same Llama-3.2-1B at 120–144 tok/s. This card is about 1.5–2.5× faster on decode and an order of magnitude faster on prefill.
• Vulkan on the same card (the backend a PC without CUDA gets): about the same as CUDA on a 1B, but 2–3× slower on an 8B (decode 19–27 vs 53–70 tok/s) and 3–4× slower on prefill.

2.11 Heat

Windows ports the DGX's Hunyuan3D heat pacer as a brake at each card's own limit (NVML's slowdown threshold, 95 °C on this RTX 3090):

• it engages on a hardware slowdown that persists for two readings, or within 8 °C of the limit;
• while engaged, every path that hands the GPU new work waits; work in flight finishes, so no answer is cut off;
• it releases after at least 8 s and 3 readings in a row 15 °C or more under the limit; no request waits more than 300 s;
• an owner can lower the line for a small case or a laptop (settings.json "heat": {"limit_c": N}), never raise it.

With a limit to enforce, the brake held the card under it and board power halved; work slowed, as it must. At the card's own limit it never engaged: 0 times in that run, and 0 times in the 3-hour run at up to 83 °C, where a single hardware-slowdown reading at 78 °C came and went without effect. In-flight work carries the temperature about 4 °C past the engage line, which at the real limit stays under the card's own 95 °C protection.

2.12 Disks

Every lane start and every rotation reads its model from disk; from a USB hard disk one switch of a 5 GB model costs about 4 minutes. The app warns at launch when the models folder is on a USB hard disk.

2.13 Energy

Board power sampled once a second through every job and integrated per job (GPU only, not the wall socket):

• A resident best-of-4 job costs about a third of a watt-hour of GPU energy. At $0.15 per kWh, a thousand such jobs cost about 5 cents.
• Rotation costs energy as well as time: the rotating fleets used about 2.4 Wh per job against 0.2–0.8 Wh for resident ones, mostly loading and unloading models.
• Idle: with lanes resident and nothing running, the card settles at 37–66 W.

2.14 One PC, one Mac, one Spark

A 24 GB gaming GPU is a very good Nightshift host for small and mid-sized fleets: it does in seconds what the 16 GB Mac does in minutes, because the whole fleet stays resident and decodes 1.5–2.5× faster. It cannot hold the 27–32B models that produce the Spark's best work, and on a PC the GPU is shared with whatever else the owner runs.

3. Strengths

• Throughput on a gaming card. Every catalog fleet that fits completes every job in seconds; 108 of 108 jobs, 93 % correct winners, the panel judging every one.
• It runs for hours without drifting. 553 of 553 jobs in 3 hours at a steady rate, lane memory flat to within 43 MB, every status sample answered.
• It adapts to a shared machine. Plans pack against the VRAM free right now; a model that would not fit beside the fleet waits rather than spilling onto the CPU; a model that would overrun the commit limit is refused by name.
• Honest refusals. Licence-gated models, a curator below its context floor, a fork from a missing checkpoint, consistency mode without a canon: each refused by name, never guessed.
• Bulk and makes at speed. 180 of 180 bulk units with one wrong field; refine fixed 6 of 6 seeded bugs; Auto's 256-attempt plan ran in under 9 minutes.
• Safe heat behaviour at no cost. A brake that held an owner-set line and never engaged in normal work.
• The release gates pass with margin, including the switch-time gate (3.3 s against 7 s).
• Installs keep the owner's state. Every install over the running app left lanes, fleets and keys byte-identical.

4. Weaknesses and limits

• The judge is the ceiling. Shown a correct candidate, the 3B judge crowns another in 19 of 42 grid jobs. Fleets that seat no capable judge (asr-factory, percept-lite: a 1B "thinking" router plus a reasoning or OCR model) end with unparseable panels.
• "Verified" is static. A generated app's verified means it parses: 10 of 12 verified, 6 worked; a missing click handler passes 3 of 3 times.
• Synthesis needs a seat. When the 8B curator cannot get room beside the fleet, most jobs go without their per-job synthesis (133 of 553 in the 3-hour run). The queue is bounded and each skipped night says why, but nothing seats the curator for them.
• A media make for weights that are not installed is retried four times over about 8 minutes before it fails.
• Rubric and canon judges still pass what they forget to check (a wrong line item) and flag some true answers (precision 0.71), as on the Mac.
• Work after a burst. After many jobs finish, the engine host stays busy with them for some minutes, and API latency rises while it does.
• Big models. 24 GB holds no 27–32B dense model beside anything else.
• The PC is shared. A game, an emulator or a large app takes VRAM, compute and commit; Nightshift adapts its plans but cannot reserve the card against the owner.
• The installer stops every llama-server.exe on the PC, not only Nightshift's.

5. What the numbers say

1. The Windows edition delivers a fleet's ability at the card's speed. With the fleet resident, a best-of-4 job is a few seconds of GPU work and a third of a watt-hour. Every catalog fleet that fits completed every job; the winners were as correct as the models allow.
2. Residency decides speed. Fleets that fit on the card finish in 3–10 s; fleets that rotate models through it finish in 9–33 s, about 21 s of which is switching. A smaller card, or a busier desktop, moves more fleets into the second group.
3. Selection, not generation, is the ceiling. The panel sees the right answer in 42 of 45 jobs that have one, and a 3B judge still picks another in 19 of them. Better judges, not more attempts, are the next gain; the Mac and the DGX reached the same conclusion.
4. It is steady over hours. Three hours of back-to-back jobs ran at a constant rate with flat memory, and a restart with work still queued kept the API answering.
5. A PC is a shared machine. The planner reads what is free right now, which kept every job alive while other programs came and went. What it cannot do is reserve the card against the owner, and it should not.

Sources and method notes

Raw data is on the bench PC under C:\NightshiftBench\; the harnesses are in nightshift-build/bench-win/. Each suite's final run:

All builds are Windows edition 1.2.1 from one source line, installed over the running app; the shipped build is 13 (setup sha256 4b7e858f…). Each suite's numbers come from its last run. The builds after it changed model rotation, memory accounting and startup behaviour, which the long runs exercised, so a repeat of an early suite on the shipped build could differ in detail (the campaign report records each change). Not measured here: the 7 catalog fleets that need 12–14B models; media renders (no image, video or music weights on this PC); a card smaller than 24 GB; a second PC; the wall-socket power of the whole machine.