Eye of the Beholder: Foundation Models Gets On-Device Vision

Hit the Lights

There’s a specific kind of feature you’ve always had to send to the cloud. Anything where the app needs to look at something — a receipt, a label, a photo — and reason about it. You reach for a vision endpoint, you eat the round trip, you pay per image, and you quietly hope the user has signal.

That whole calculation just changed. The on-device Foundation Models framework now takes images alongside text in a single prompt. You hand the model a photo, you ask a question grounded in what’s in it, and the answer comes back from the Neural Engine. No network. No per-token bill. No “this feature requires an internet connection” empty state.

That last part is the real unlock. Not that the model can see — plenty of models can see. It’s that it can see here, on the device, inside the trust boundary of the app. The photo never leaves. For anything remotely personal — and a photo of the thing sitting on your desk is personal — that’s the difference between a feature you can ship and a privacy review you’d rather not have.

I wired this into VinylCrate over a weekend, and it’s the first time a “point the camera at a thing” feature felt like it belonged in a small indie app instead of a startup with a cloud budget. Here’s how the pieces actually fit.

Eye of the Beholder

Foundation Models is multimodal now — the prompt can carry an image segment alongside the instruction text. Everything else is unchanged from the text-only API you already know: you have a LanguageModelSession, you call respond, you get content back. The image is the only new part of the shape.

A quick honesty note before the code: the exact spelling of the image-attachment type is the part of this API I’d double-check against the SDK you’re building with, because it’s the newest surface and the most likely to have been renamed between betas. The shape below is what a working iOS dev will recognize — a Prompt built from an instruction plus an image segment — and that’s what matters for wiring your own version. Treat the image type name as the one line to verify, not the structure.

import FoundationModels
import UIKit

func describeSleeve(_ image: UIImage) async throws -> String {
    let session = LanguageModelSession(
        instructions: "You are a vinyl cataloging assistant. Be precise and terse."
    )

    let prompt = Prompt {
        "Describe the album sleeve in this photo — artist, title, and any label branding you can see."
        PromptImage(image)
    }

    return try await session.respond(to: prompt).content
}

That’s a description. Useful, but a String is a dead end when the next thing you do is populate a form. This is where guided generation carries its weight. Instead of parsing prose, you tell the model the shape you want and let it fill the struct.

@Generable(description: "Structured details read from a vinyl record sleeve")
struct SleeveReading: Sendable {
    @Guide(description: "Primary recording artist or band")
    var artist: String

    @Guide(description: "Album or release title")
    var title: String

    @Guide(description: "Record label, if visible")
    var label: String?

    @Guide(description: "Four-digit release year, if printed", .range(1900...2100))
    var year: Int?
}

And the call site barely changes — you’re just asking the session to generate that type from the same image prompt:

let reading = try await session.respond(
    to: prompt,
    generating: SleeveReading.self
).content

Now you’ve got a typed value out of a photograph, entirely on-device. For a lot of use cases you could stop right here.

But vinyl has a wrinkle, and the wrinkle is the whole reason this post exists.

Sad but True

The Foundation Models vision stack is genuinely good at understanding a scene. Ask it what’s on the sleeve and it’ll tell you it’s a jazz record with a blue-and-white cover and a saxophone. That’s comprehension.

What it is not built to do is transcribe a nine-character catalog number etched in tiny type across the bottom of the jacket, or read the matrix runout code stamped into the dead wax. Those strings are the entire point of cataloging a record — BLP 1577, SR-90521, the pressing variant hiding in a runout — and they’re exactly the kind of dense, low-context, high-precision text a language model will happily hallucinate one character off. “Close” is worthless here. A catalog number that’s one digit wrong points at a different pressing worth a different amount of money.

So don’t ask the LLM to be an OCR engine. Give it one.

This is the pattern I keep coming back to with on-device models: let the LLM do the reasoning it’s good at, and hand it a tool for the deterministic work it’s bad at. The model understands the sleeve. Vision framework reads the exact text. The model gets ground truth handed to it mid-generation instead of guessing.

Seek & Destroy

Foundation Models’ tool calling is the join. You define a Tool, the session decides when to call it, and the recognized text flows back into the model’s context so its structured answer is grounded in real characters rather than its own reading of blurry type.

The tool wraps Vision’s modern async OCR. On iOS 18+ you get RecognizeTextRequest, which drops the completion-handler ceremony of the old VNRecognizeTextRequest for a clean await. (The completion-handler API still works if you’re supporting older OS versions — you’d bridge it with a continuation — but on an iOS 26 floor there’s no reason to.)

import Vision

struct SleeveOCRTool: Tool {
    let name = "readSleeveText"
    let description = "Extract exact printed text — catalog numbers, track listings, \
label copy — from the record sleeve image."

    // The image lives on the tool, not in the arguments — the model can't
    // pass a CGImage through a generated struct, and it shouldn't need to.
    let image: CGImage

    @Generable
    struct Arguments: Sendable {
        @Guide(description: "What kind of text to prioritize, e.g. 'catalog number' or 'track listing'")
        var focus: String
    }

    func call(arguments: Arguments) async throws -> String {
        var request = RecognizeTextRequest()
        request.recognitionLevel = .accurate
        request.usesLanguageCorrection = false  // catalog codes aren't words

        let observations = try await request.perform(on: image)
        let lines = observations.compactMap { $0.topCandidates(1).first?.string }

        guard !lines.isEmpty else { return "No text found on the sleeve." }
        return lines.joined(separator: "\n")
    }
}

Two decisions in there earn their comments. The image rides on the tool rather than the Arguments, because the model has no business — and no ability — to marshal a CGImage through generated arguments; it only needs to decide that it wants text and what to focus on. And usesLanguageCorrection is off on purpose. Language correction is a feature when you’re reading sentences and a liability when you’re reading SR-90521, which spell-check would love to turn into something that looks more like English.

Then you attach the tool at session construction, and the model calls it on its own when it decides it needs precise text:

let session = LanguageModelSession(
    tools: [SleeveOCRTool(image: cgImage)],
    instructions: """
        You are a vinyl cataloging assistant. Identify the release from the sleeve image.
        When you need exact printed text — catalog numbers, track listings, matrix codes — \
        call readSleeveText rather than reading the characters yourself.
        """
)

That instruction is doing real work. Without the nudge, the model tends to trust its own eyes and skip the tool. Tell it explicitly that exact strings are the tool’s job and it leans on the tool for precisely the text it’s worst at.

Master of Puppets

Here’s how the whole thing lands as a feature. In VinylCrate, you point the camera at an album jacket, tap scan, and the review sheet comes back pre-filled — artist, title, label, catalog number, year — with the OCR-verified strings already slotted in. You confirm or fix, then search Discogs to match the pressing. The scan turns a two-minute typing chore into a two-second glance.

The model is pulling the strings here, and that’s the point. It orchestrates: it reads the scene, decides it needs exact characters, calls the OCR tool, folds the result back into its own reasoning, and hands you a finished struct. You wired the tool; the model decides when to reach for it.

The generable output for that is a little richer than the earlier reading, because now it can carry the tool-verified fields:

@Generable(description: "A record release identified from its sleeve, ready to catalog")
struct ScannedRelease: Sendable {
    @Guide(description: "Primary recording artist or band")
    var artist: String

    @Guide(description: "Album or release title")
    var title: String

    @Guide(description: "Record label")
    var label: String?

    @Guide(description: "Catalog number exactly as printed — copy from OCR, do not guess")
    var catalogNumber: String?

    @Guide(description: "Release year", .range(1900...2100))
    var year: Int?
}

The @MainActor view model that drives the scan screen owns the session and stays on the main actor because it’s feeding SwiftUI directly. The ScannedRelease is a Sendable value type, so it crosses back from generation to the UI without a fight from the compiler.

@MainActor
@Observable
final class ScanViewModel {
    var release: ScannedRelease?
    var state: State = .idle

    enum State: Sendable { case idle, scanning, failed(String) }

    func scan(_ image: CGImage) async {
        guard case .available = SystemLanguageModel.default.availability else {
            state = .failed("On-device scanning isn't available on this device.")
            return
        }

        state = .scanning
        let session = LanguageModelSession(
            tools: [SleeveOCRTool(image: image)],
            instructions: "Identify the release. Use readSleeveText for exact catalog numbers and track listings."
        )

        let prompt = Prompt {
            "Identify this vinyl record from its sleeve and fill in the catalog details."
            PromptImage(image)
        }

        do {
            release = try await session.respond(
                to: prompt,
                generating: ScannedRelease.self
            ).content
            state = .idle
        } catch let error as LanguageModelSession.GenerationError {
            state = .failed(userMessage(for: error))
        } catch {
            state = .failed("Couldn't read the sleeve. Try a clearer photo.")
        }
    }
}

The camera capture produces a CGImage, the view model runs the session, and the resulting ScannedRelease maps straight onto the review sheet’s fields. The sheet is still a form — the user is still in control — the scan just fills the tedious parts and gets the catalog number right.

Nothing Else Matters

This is the part I’d have wanted someone to tell me before I shipped it.

Check availability, and mean it. SystemLanguageModel.default.availability isn’t a formality. On a device that isn’t Apple Intelligence-eligible, or with the model still downloading, the whole feature is unavailable — not degraded, gone. The scan button needs a real fallback: manual entry, which is what the app did before this feature existed anyway. On-device AI is an accelerant, not a dependency. Build it so the app is whole without it.

Blurry input is your most common failure, not model refusal. People photograph records in bad light at bad angles. The OCR tool returns “No text found,” the model does its best from the visual alone, and the catalog number comes back nil. That’s the honest outcome — a partial fill with the precise field left empty is better than a confident wrong number. Design the form to celebrate a partial scan instead of treating it as an error.

Handle refusal separately from failure. A guardrail violation or a .refusal is a different branch than a decoding failure, and the user message should reflect that. Most of the time you’ll collapse them into “try a clearer photo,” but keep the branches distinct in code so you can log — and test — what’s actually happening.

Latency is real but local. A vision prompt with a tool round trip is not instant — you’re looking at a couple of seconds on current hardware, more on the first call before the session warms. Prewarm the session when the user opens the camera, show honest progress, and don’t block the whole UI on it.

The concurrency shakes out clean if you let it. @Generable output types are Sendable value types. The Tool conformer is Sendable because it holds an immutable CGImage and nothing mutable. The session lives on a @MainActor view model because it drives the UI. Get those three right and Swift 6 has nothing to complain about — no @unchecked, no unsafe escape hatches, just isolation that matches how the data actually moves.

The thing I keep sitting with is how quietly this shifts what’s buildable. A year ago, “point your camera at a record and it fills in the catalog number” was a cloud feature with a cost model and a privacy story. Now it’s a weekend in a hobby app, and the photo of your record collection never leaves your phone.

Is the model’s own reading perfect? No. That’s why the OCR tool exists. Is the combination better than either piece alone? Every time.