Local model throughput is not a number you read off the side of a box. The parameter count tells you almost nothing about how a model behaves on your Mac. Quantization changes the arithmetic. KV-cache pressure climbs as the context fills. And the moment the thermal headroom runs out, the silicon throttles and your tokens-per-second falls off a cliff — quietly, without an error, without a log line.
You can’t feel any of that from a CLI that prints a token count at the end. A final number is the average of a story you didn’t get to watch. The interesting part is the shape of the curve while it runs. So I built a harness to watch it: ModelBench, a macOS 26 app that runs eval suites against local backends, captures live telemetry — tokens/sec, time-to-first-token, resident memory, thermal state — and charts it in real time while persisting every run for comparison.
But the app is the easy part. The part worth writing about is the concurrency design at its center: merging two independent, live async streams — inference and telemetry — into one ordered event stream, under a single structured task group, with zero data races and zero Task.detached. Swift 6 strict concurrency from the first line. That’s the post.
Two streams that need to become one
A benchmark run is two things happening at once.
The first is inference: prompts going to a backend, tokens streaming back. Sequential by nature — you ask, you wait, you read the chunks. The second is telemetry: the system sampling itself on a fixed cadence, completely indifferent to whatever the model is doing. One is event-driven and bursty. The other is a metronome ticking every 500ms.
The UI wants one thing: an ordered stream of events it can fold into SwiftUI state. A token rate update here, a completed eval case there, a final result at the end. The runner’s whole job is to take those two unsynchronized sources and braid them into a single AsyncThrowingStream<RunEvent, Error> — one timeline out of two clocks that never agreed to keep time together.
The naive instinct is to spin up a detached task for telemetry, let it write into some shared dictionary of “current stats,” and have the inference loop read from it. That’s a data race wearing a lab coat. Two isolation domains, shared mutable state, no synchronization — the compiler will let you do it only if you lie to it with @unchecked Sendable, and then you own every bug that follows.
The honest version puts all the run state on one actor and lets structured concurrency do the proof.
The runner is an actor, and that’s the whole trick
BenchmarkRunner is an actor. Every piece of mutable run state — the token counter, the last-sample bookkeeping, the time-to-first-token latch, the collected results — lives on it and nowhere else.
actor BenchmarkRunner {
enum RunEvent: Sendable {
case telemetry(TelemetrySample)
case caseCompleted(EvalResult)
case finished(BenchmarkRun)
}
// Run-scoped state (reset at the start of each run).
private var generatedTokens = 0
private var lastSampleTokens = 0
private var lastSampleInstant: ContinuousClock.Instant?
private var firstTokenLatency: Duration?
private var collectedSamples: [TelemetrySample] = []
private var collectedResults: [EvalResult] = []
// …
}
The public entry point hands back a stream and kicks off the work. Note the onTermination hook — if the consumer tears down (the user hits cancel, the view disappears), the run task gets cancelled. No orphaned work:
func run(
suite: any EvalSuite,
backend: any ModelBackend,
model: String,
options: GenerationOptions = .benchmark
) -> AsyncThrowingStream<RunEvent, Error> {
AsyncThrowingStream { continuation in
let task = Task {
await self.execute(/* … */, continuation: continuation)
}
continuation.onTermination = { _ in task.cancel() }
}
}
Now the heart of it. Inside execute, a single withThrowingTaskGroup runs the telemetry consumer concurrently with the sequential inference loop:
try await withThrowingTaskGroup(of: Void.self) { group in
let systemStream = await monitor.start(interval: .milliseconds(500))
// Child: enrich each system sample with current tok/s and emit it.
group.addTask {
for await sys in systemStream {
let sample = await self.makeTelemetrySample(from: sys, now: clock.now)
continuation.yield(.telemetry(sample))
}
}
// Parent: run eval cases sequentially.
for evalCase in suite.cases {
try Task.checkCancellation()
let caseStart = clock.now
var output = ""
for try await chunk in backend.stream(prompt: evalCase.prompt, model: model, options: options) {
try Task.checkCancellation()
if chunk.hasGeneratedText {
registerGeneratedToken(elapsed: { clock.now - caseStart })
}
output += chunk.contentDelta
}
let result = await scorer.score(evalCase, output: output)
collectedResults.append(result)
continuation.yield(.caseCompleted(result))
}
await monitor.stop() // ends systemStream, letting the child return
group.cancelAll()
}
Look at what makes this safe, because it’s subtle and it’s the entire reason the design holds.
The inference loop is the parent. It suspends on every awaited chunk — for try await chunk in backend.stream(...). Each suspension is a yield point. While the parent is parked waiting on the network, the telemetry child is free to run: it pulls a SystemSample, calls back into the actor via makeTelemetrySample, mutates the counters, and yields a RunEvent.
Both the parent and the child touch the same mutable state — generatedTokens, lastSampleInstant, the rest. But every one of those touches is an actor-isolated method call. The actor serializes them. The parent increments generatedTokens while reading a chunk; the child reads generatedTokens to compute a rate. They interleave constantly, and they cannot race, because the actor guarantees one at a time. One actor, pulling every string — nothing touches that state without going through it. The compiler proves it. I didn’t write a single lock.
This is what Swift 6 buys you when you stop fighting it. Pick the right isolation and the data-race freedom is a theorem, not a hope.
The runner owns the math; the monitor owns nothing
There’s a design boundary here I care about more than the code.
TelemetryMonitor knows nothing about inference. It doesn’t know what a token is. It emits raw SystemSample values — a timestamp, a memory reading, a thermal state — on a timer, and that’s the whole of its job. It’s reusable, independently testable, and completely decoupled from whatever is generating load.
The runner is where token throughput gets computed. When a system sample arrives, makeTelemetrySample is the only place that knows the current token count, so it’s the only place that can compute an instantaneous rate — the delta since the previous sample, not a run-wide average:
private func makeTelemetrySample(from sys: SystemSample, now: ContinuousClock.Instant) -> TelemetrySample {
let tokensPerSecond: Double
if let last = lastSampleInstant {
let elapsed = (now - last).seconds
let delta = generatedTokens - lastSampleTokens
tokensPerSecond = elapsed > 0 ? max(0, Double(delta) / elapsed) : 0
} else {
tokensPerSecond = 0
}
lastSampleInstant = now
lastSampleTokens = generatedTokens
let sample = TelemetrySample(
timestamp: sys.timestamp,
tokensPerSecond: tokensPerSecond,
timeToFirstToken: firstTokenLatency,
memoryMB: sys.memoryMB,
thermalState: sys.thermalState
)
collectedSamples.append(sample)
return sample
}
Instantaneous rate is what makes thermal throttling visible. A run-wide average smears the cliff into a gentle slope and hides exactly the thing you opened the app to see. Sampling the delta every 500ms means when the silicon throttles, you watch the throughput fade to black on the chart while it’s happening — not in a summary you read after the damage is done.
Real telemetry, public APIs only
The memory number is the one Activity Monitor calls “Memory” — the physical footprint, phys_footprint, read through the mach task_info call. No estimates, no resident_size approximation that drifts from what the OS actually reports:
static func residentMemoryMB() -> Double {
var info = task_vm_info_data_t()
var count = mach_msg_type_number_t(
MemoryLayout<task_vm_info_data_t>.size / MemoryLayout<natural_t>.size
)
let kr = withUnsafeMutablePointer(to: &info) { pointer in
pointer.withMemoryRebound(to: integer_t.self, capacity: Int(count)) { rebound in
task_info(mach_task_self_, task_flavor_t(TASK_VM_INFO), rebound, &count)
}
}
guard kr == KERN_SUCCESS else { return 0 }
return Double(info.phys_footprint) / (1024 * 1024)
}
Thermal state is ProcessInfo.thermalState, mapped onto a domain enum. I went back and forth on this one. There are richer thermal signals on Apple Silicon if you’re willing to read the SMC directly — but those are private, brittle across hardware, and a fast way to ship something that breaks on the next Mac. The public four-state ladder — nominal, fair, serious, critical — is coarse, but it’s stable, and it tells you the one thing that matters for a benchmark: is the machine throttling right now. I’ll take honest and coarse over precise and fragile every time.
Both reads are static and pure, which keeps the sampling loop trivial and the functions testable without an actor in sight.
Two backends, one protocol
ModelBench drives two local servers, and they don’t speak the same language. The protocol papers over the difference with a single value-typed surface:
protocol ModelBackend: Sendable {
var kind: BackendKind { get }
var baseURL: URL { get }
func availableModels() async throws -> [String]
func stream(
prompt: String,
model: String,
options: GenerationOptions
) -> AsyncThrowingStream<InferenceChunk, Error>
}
OllamaBackend talks the native /api/chat protocol rather than Ollama’s OpenAI-compatible shim — deliberately, because the native protocol hands back authoritative counts: eval_count and eval_duration on the terminal chunk. It parses NDJSON, one JSON object per line:
for try await line in bytes.lines {
try Task.checkCancellation()
guard !line.isEmpty,
let data = line.data(using: .utf8),
let chunk = try? JSONDecoder().decode(ChatChunk.self, from: data)
else { continue }
let isDone = chunk.done ?? false
continuation.yield(
InferenceChunk(
contentDelta: chunk.message?.content ?? "",
reasoningDelta: chunk.message?.thinking ?? "",
finalCompletionTokens: isDone ? chunk.evalCount : nil,
isFinal: isDone
)
)
if isDone { break }
}
MLXBackend speaks OpenAI-compatible SSE — the protocol mlx_lm.server exposes — parsing data: lines and watching for the [DONE] sentinel. Different wire format, identical InferenceChunk out the other side.
Every backend stream is a backpressure-safe AsyncThrowingStream with the same onTermination discipline: cancel the consumer and the underlying URLSession task gets torn down with it. The runner doesn’t know or care which backend it’s driving. It sees chunks.
Grading SwiftUI with a second model
Not everything a model produces can be scored with a regex. Instruction-following cases — count the bullets, check for a forbidden word, validate the JSON — score deterministically. But “write me a SwiftUI master-detail view” has no exact-match answer.
So ModelBench grades those with a second model. JudgeClient sends the candidate output and a rubric to a judge model and asks for a strict JSON verdict:
struct Verdict: Sendable, Equatable {
let pass: Bool
let score: Double
let reason: String
}
The rubrics are opinionated about modern SwiftUI — reward @Observable, NavigationStack, .foregroundStyle, .task; penalize NavigationView, ObservableObject, .foregroundColor, Task.detached inside onAppear. Which raises the only question that matters once a model is grading a model: how much do you trust the judge, and how do you keep it honest when nobody’s checking its work? Testing output that won’t give the same answer twice is already its own discipline — a model grading a model is that with the floor removed — justice for all, handed down by a juror you can’t cross-examine. That’s a whole post on its own, and it’s the next one.
Persistence is its own isolation domain
Runs need to survive the app. They go into SwiftData — but SwiftData on the main actor while you’re charting a live run is how you drop frames. So the store is a @ModelActor:
@ModelActor
actor BenchmarkStore {
func save(_ run: BenchmarkRun) throws {
modelContext.insert(RunRecord(run))
try modelContext.save()
}
func allRuns() throws -> [BenchmarkRun] {
let descriptor = FetchDescriptor<RunRecord>(
sortBy: [SortDescriptor(\.startedAt, order: .reverse)]
)
return try modelContext.fetch(descriptor).map { $0.toDomain() }
}
// …
}
All SwiftData access serializes on its own actor-bound ModelContext, off the main actor entirely. And it never leaks @Model types across the boundary — it converts to and from plain Sendable domain values, so the rest of the app talks in BenchmarkRun, not in persistence records. The view model is @MainActor @Observable, consumes the RunEvent stream, folds telemetry into a sliding 120-sample window for the chart, and hands the finished run to the store. Three isolation domains — main actor for UI, the runner for orchestration, the store for persistence — and the compiler keeps them from stepping on each other.
Once runs are persisted, comparing them is just a fetch — two runs side by side, throughput against pass rate against peak memory:
The sandbox wall
One eval suite I had to leave half-built, and I want to be honest about why.
HumanEval scores Python by running it — you generate code, you execute the test cases, you count passes. ModelBench has the suite and the prompts. What it can’t do, in the shipped build, is run the code: the macOS app sandbox blocks spawning python3, and a benchmark harness that silently fakes a score is worse than useless. So those cases report as execution-disabled, explicitly, rather than lying with a green checkmark.
You can enable it — drop com.apple.security.app-sandbox from the entitlements and regenerate. But understand what that means: you’re running untrusted model output on your machine with no fence around it. That’s a real security tradeoff, and it should be a deliberate choice, not a default I made for you. Surfacing the wall and labeling it beats pretending it isn’t there.
What’s next
The concurrency core is done and it holds. Two things come after it, and they’re the ones I actually want to write: a deep dive on the LLM-as-judge — how much you can trust a model grading another model’s code, and what it takes to keep that grader honest — and the first real numbers, MLX versus Ollama throughput on the same prompts, same hardware, charted live with the thermal cliff in frame.
ModelBench targets macOS 26+ and is written against Swift 6 strict concurrency throughout — no @unchecked Sendable, no Task.detached, no locks. The full source is up on GitHub — read the runner, file an issue, or tear the concurrency apart and tell me where it breaks.
Two streams, one task group, zero races. That’s the part worth keeping.