Sad but True: Localized App Store Screenshots That Verify Themselves

I found this one standing in line for coffee, which is not where I do my best QA.

Someone asked what I’d been building, so I pulled up VinylCrate on the App Store — with my region still flipped to Spain from an hour earlier, when I’d been poking at a price tier. The screenshots loaded. Framed, captioned, nicely composed, and every last one of them in English.

The Spanish translation was fine. It had been fine for weeks. What had broken was the screenshot pipeline: it never applied the locale at all, the app quietly fell back to its development region, and the images that came out the other end looked completely correct to anybody who wasn’t reading them.

That’s the failure mode I want to talk about — partly because it’s a nasty one on its own, and partly because it went on to beat me twice more before I was finished, once in a way I’m fairly sure I’d never have caught by looking.

Sad but True — Why a Wrong-Language Screenshot Looks Right

Here’s the thing about a screenshot that renders in the wrong language: it doesn’t look broken. It looks like a screenshot.

Nothing about it trips a check. No crash, no red diff, no missing-asset placeholder, not even a Localizable.strings key leaking through as screenshot.caption.library to give the game away. A wrong-language screenshot is a correct rendering of the wrong thing — every pixel doing its job, nav bar aligned, type fitting the space it was given. The only signal is semantic, which means the only detector is somebody who reads that language, looking at the image at full size, at the exact moment it matters.

Now count them. VinylCrate’s 4.0 set is eight panels, Apple wants a 6.9-inch iPhone set and a 13-inch iPad set, and we ship six languages — en es fr de ja zh-Hans. That’s 8 × 2 × 6 = 96 images, and App Store Connect hands them back to you as a scrollable strip of thumbnails maybe 90 points wide. You’re not reading kanji at 90 points. You’re definitely not telling Spanish from English at 90 points, because at that size both of them are just “some Latin script in roughly the right place.”

The scripts that are obviously different are the ones you’d catch anyway. Japanese, Simplified Chinese — one glance and you know. It’s German and French and Spanish that slip past, because they share an alphabet with your development region and a decent chunk of their proper nouns. The locales most likely to break silently are the ones you’re least equipped to notice, which is a fun property for a bug to have.

So manual review here isn’t just slow, it’s the wrong instrument. Ninety-six images isn’t too many to look at; I’ve reviewed worse. The problem is that “look at it” can’t answer the question. The question is a comparison: is the German panel byte-identical to the English one? If the answer is ever yes, something upstream lied to you.

That’s trivial for a machine and close to impossible for a person, so I stopped reviewing and wrote the check instead. It’s 146 lines of bash, the interesting part is shasum -a 256, and it has since caught two real bugs.

Master of Puppets — Seeding an App That Can’t Drift

A screenshot harness is a determinism problem wearing a UI-testing costume. Everything you’re about to hash has to be a pure function of the inputs, or the hashing is worthless. Two panels that differ because a relative timestamp ticked over are two panels whose hashes tell you nothing.

VinylCrate’s fixture is a ScreenshotSeed enum that builds an in-memory SwiftData container and populates it:

static func makeInMemoryContainer() -> ModelContainer? {
    do {
        let container = try ModelContainer(
            for: SharedModelContainer.schema,
            configurations: [ModelConfiguration(
                schema: SharedModelContainer.schema,
                isStoredInMemoryOnly: true,
                allowsSave: true,
                cloudKitDatabase: .none
            )]
        )
        let context = ModelContext(container)
        try populate(context)
        return container
    } catch {
        logger.error("❌ Screenshot seed container failed: \(error.localizedDescription)")
        assertionFailure("Screenshot seed container failed: \(error)")
        return nil
    }
}

isStoredInMemoryOnly: true and cloudKitDatabase: .none are doing the real work: no store on disk to carry state between runs, and no sync to race with. The seeded library is twelve records (five Metallica, seven Pearl Jam) with fixed prices, fixed favorites, and fixed artwork.

The clock is frozen by construction rather than injected:

static let referenceDate = Date(timeIntervalSince1970: 1_772_500_000)

Every seeded row hangs off that constant. Play events land at referenceDate.addingTimeInterval(-hoursBack * 3600), copy photos at -Double(3 - index) * 3600, the dead wax scan three days back. Anything that renders “2 days ago” renders it the same way every run, because the data is anchored, not the formatter. Photo IDs come from DeterministicUUID.make(namespace:name:) for the same reason.

Two more pieces close the loop. The seed is #if DEBUG-gated so it cannot possibly activate in a release build:

static func isActive(in arguments: [String] = ProcessInfo.processInfo.arguments) -> Bool {
    #if DEBUG
    arguments.contains(launchArgument)
    #else
    false
    #endif
}

And one network call gets stubbed, because Discogs’ /collection/value needs a real OAuth token the seeded launch never obtains and would otherwise 401 into an empty Insights panel:

static let stubCollectionValueTransport: DiscogsClient.Transport = { request in
    guard
        let url = request.url,
        url.path.hasSuffix("/collection/value"),
        let response = HTTPURLResponse(url: url, statusCode: 200, httpVersion: nil, headerFields: nil)
    else {
        return try await URLSession.shared.data(for: request)
    }
    return (stubCollectionValueBody, response)
}

The stubbed figure is deliberately a little above the seed’s $310.45 total invested, so the panel reads as plausible appreciation instead of an arbitrary number. Marketing screenshots have to survive a skeptical reader, not just a hash.

Then there’s the part I did not predict. The launch is --seed-screenshots --disable-tips, and even with an in-memory store and TipKit silenced, one piece of state kept leaking across runs: @AppStorage("importPromptState"). It lives in user defaults, which the in-memory container knows nothing about, and it would present the import sheet on top of a capture in a reused simulator.

The fix is a bigger hammer than I wanted and I’ve made peace with it:

xcrun simctl shutdown "$device" 2>/dev/null || true
xcrun simctl erase "$device"
xcrun simctl boot "$device"
xcrun simctl status_bar "$device" override \
  --time "9:41" --batteryState charged --batteryLevel 100 \
  --cellularBars 4 --wifiBars 3 --dataNetwork wifi

Erase, boot, pin the status bar, twelve times per full run. It costs real minutes and I’d rather spend them than debug a capture that inherited something from the run before it. The status bar override does double duty here. App Store screenshots are supposed to look that way, and a clock left running would change every hash on every run.

Wherever I May Roam — Why -testLanguage Fails and simctl Doesn’t

Now the part that started all of this.

xcodebuild has a -testLanguage flag. It’s right there in the man pages, it’s the obvious tool for the job, and on the Xcode 27 toolchain this project is pinned to (27A5252f), it does not reliably reach the app under test. Read the promise closely: it “overrides the setting for the test action of a scheme in a workspace.” That’s a claim about the scheme’s test action, not about the process your UI test actually launches, and the gap between those two is where six languages go missing. I confirmed it the boring way: ran the capture with -testLanguage ja and got identical English UI out the other end. No warning, no error, exit 0. I haven’t filed feedback on it yet, which I should.

Which is the whole problem in miniature: a flag that silently doesn’t apply, sitting in a pipeline whose output is images nobody reads closely.

What actually works is setting the simulator’s own global domain before the test launches the app:

xcrun simctl spawn "$device" defaults write -g AppleLanguages -array "$lang"
xcrun simctl spawn "$device" defaults write -g AppleLocale -string "$(locale_for "$lang")"

-array "$lang" matters. AppleLanguages is an array; write a bare string into that slot and it isn’t an error, it’s just a value the localization machinery ignores while the bundle quietly resolves to CFBundleDevelopmentRegion. This is the same trap as the old -AppleLanguages es versus -AppleLanguages "(es)" parenthesis gotcha, wearing different clothes.

The language-to-locale mapping is a plain case, and it’s explicit on purpose:

locale_for() {
  case "$1" in
    en) echo "en_US" ;;
    es) echo "es_ES" ;;
    fr) echo "fr_FR" ;;
    de) echo "de_DE" ;;
    ja) echo "ja_JP" ;;
    zh-Hans) echo "zh_Hans_CN" ;;
    *) echo "en_US" ;;
  esac
}

zh-Hans is the structural outlier, the only one using a script subtag, which is why it’s written out by hand rather than derived. A clever "${lang}_$(echo "$lang" | tr '[:lower:]' '[:upper:]')" would produce zh-Hans_ZH-HANS, and you’d find out from a screenshot.

I still pass -testLanguage "$lang" to xcodebuild alongside the defaults write. Belt and suspenders: it costs nothing, and if a future toolchain fixes the flag, the two mechanisms agree rather than fight.

One more trap, and it’s pure paperwork that will waste an afternoon: this project’s build default is iPhone 17 Pro, which is 6.3 inches, which is the wrong size. App Store Connect wants 6.9-inch, 1320 × 2868, and the iPad Pro 13-inch at 2064 × 2752. Apple scales every other size from those two, so the whole matrix is two devices, and both have to be named explicitly:

DEVICES=("iPhone 17 Pro Max" "iPad Pro 13-inch (M5)")

Get that wrong and every image is beautiful, localized, correct, and rejected on upload.

Seek & Destroy — Catching Duplicate Screenshots with shasum

Everything up to here is plumbing. This is the part that actually fixed my problem.

The assertion is one sentence: no two languages may produce a byte-identical panel. Write it down that way and it stops being something a human has to notice and becomes something the script enforces.

assert_language_hashes_distinct() {
  local slug="$1"
  local panel hashfile dup dup_langs lang f
  for panel in "${PANELS[@]}"; do
    hashfile=$(mktemp)
    for lang in "${LANGS[@]}"; do
      f="$OUT/$slug/$lang/panel-$panel.png"
      shasum -a 256 "$f" | awk -v lang="$lang" '{print $1, lang}' >> "$hashfile"
    done
    dup=$(awk '{print $1}' "$hashfile" | sort | uniq -d | head -1)
    if [ -n "$dup" ]; then
      dup_langs=$(awk -v h="$dup" '$1==h {print $2}' "$hashfile" | tr '\n' ' ')
      echo "FAIL $slug panel-$panel: byte-identical across languages: $dup_langs (hash $dup)" >&2
      rm -f "$hashfile"
      exit 1
    fi
    rm -f "$hashfile"
  done
}

That’s a temp file and sort | uniq -d rather than an associative array, because the system bash this script’s shebang resolves to is 3.2.57 and predates declare -A — still what /bin/bash is on macOS, still going to be true next year. Inelegant, but it runs everywhere without me installing anything.

Exact bytes is the right gate here, and the reason matters, because perceptual hashing is the reflexive answer and it’s wrong for this. These images get captured minutes apart, on the same machine, from the same build, with the status bar pinned. There’s no antialiasing drift to absorb, nothing cross-toolchain, no golden image checked in last spring that I’m hoping still matches. In that setting SHA-256 has zero false positives: if two renders are byte-identical they are the same image. A perceptual hash would only buy me a threshold to tune and a new way to be wrong.

So that guard shipped, went green, and was still lying to me.

It compares the same panel number across languages, which makes it structurally blind to a different failure: the wrong screen captured within one language. Which is exactly what was happening. testPanel06MyPhotosGallery was landing on the Dead Wax section instead of the My Photos gallery, producing an image byte-identical to panel-02 — on iPhone, in every language. Every language had the same wrong screen, so the cross-language guard saw six distinct-per-panel hashes and reported success.

The generalization is four lines of different bookkeeping:

assert_panels_pairwise_distinct() {
  local dest="$1"
  local panel hashfile dup dup_panels f
  hashfile=$(mktemp)
  for panel in "${PANELS[@]}"; do
    f="$dest/panel-$panel.png"
    shasum -a 256 "$f" | awk -v panel="$panel" '{print $1, panel}' >> "$hashfile"
  done
  dup=$(awk '{print $1}' "$hashfile" | sort | uniq -d | head -1)
  if [ -n "$dup" ]; then
    dup_panels=$(awk -v h="$dup" '$1==h {print $2}' "$hashfile" | tr '\n' ' ')
    echo "FAIL $dest: panels byte-identical: $dup_panels (hash $dup)" >&2
    rm -f "$hashfile"
    exit 1
  fi
  rm -f "$hashfile"
}

It’s the same primitive rotated ninety degrees. One hashes a panel across languages, the other hashes a language’s panels against each other, and the failures they catch don’t overlap at all. Neither one subsumes the other, which took me an embarrassing minute to be sure of.

When you write a duplicate check, ask what axis it’s blind to. A guard that compares along one dimension of a matrix will sail straight past a bug that varies along the other, and it’ll print ok while it does.

The Thing That Should Not Be — Guards That Test Themselves

A guard that has never fired is a guard you’re trusting on faith. Mine were written to catch a bug that was already fixed, so a normal run proves nothing about them — a for loop with a typo in the path would also sail through without failing. It’s the same discipline as writing a test that fails first, just applied to a shell script instead.

So both guards have a dispatch mode whose entire job is to fail:

if [ "${1:-}" = "--verify-hash-guard" ]; then
  assert_language_hashes_distinct "$2"
  echo "guard did not fire (unexpected)"
  exit 1
fi

if [ "${1:-}" = "--verify-panel-guard" ]; then
  assert_panels_pairwise_distinct "$2"
  echo "guard did not fire (unexpected)"
  exit 1
fi

Follow the control flow there. If the guard fires it prints FAIL and exits 1 from inside the function, so the echo never runs. If the guard doesn’t fire, control falls through to a line that says so and exits 1 anyway. Nothing in either branch can exit 0. You point it at deliberately corrupted input and you get a failure either way — the only question is which message comes back, and that’s the whole design.

Verifying the language guard meant copying an en panel over its ja counterpart, running --verify-hash-guard, checking that the FAIL line named both languages, and putting the file back. The panel guard I got to verify twice: once by duplicating panel-04 over panel-07 in a known-clean capture, and once against the real pre-fix iPhone/en capture that was still sitting on disk, which correctly flagged panels 02 and 06.

That last one was luck, and it’s worth stealing as a habit. The bad output hadn’t been cleaned up yet, so I could test the guard against the actual failure it was written for instead of a synthetic imitation of it. Next time you fix a capture bug, hang onto the broken files until the guard that’s supposed to catch them has been run against them.

Whiplash — XCUITest, .exists, and What .isHittable Fixes

The XCUITest side is five methods, one per panel, and there is nothing clever in any of them. Launch seeded, tap along stable identifiers, assert the thing you came for, attach:

private func launchSeeded() -> XCUIApplication {
    let app = XCUIApplication()
    app.launchArguments += ["--seed-screenshots", "--disable-tips"]
    app.launch()
    return app
}

private func capture(_ app: XCUIApplication, as name: String) {
    let attachment = XCTAttachment(screenshot: app.windows.firstMatch.screenshot())
    attachment.name = name
    attachment.lifetime = .keepAlways
    add(attachment)
}

Every tap is on an accessibility identifier, never a localized string. Obvious in hindsight; not obvious the first time you write app.buttons["Settings"].tap() and watch it pass in exactly one of six runs.

Three things cost me real time.

.exists does not imply visible. This was half the panel-06 bug. SwiftUI keeps off-screen views resident in a plain VStack inside a ScrollView, so a section scrolled well past the top still reports exists == true. The assertion passed while the camera was pointed somewhere else entirely. Assert .isHittable when what you mean is “on screen and photographable.”

An accessibility identifier on a container can leak, and override. CopyPhotosSection had .accessibilityIdentifier("copyPhotos.section") on its outer VStack. Because that stack contains interactive buttons, SwiftUI doesn’t collapse it into a single accessibility element; the identifier landed on five separate descendants instead, and (confirmed by direct device inspection) silently overrode a descendant’s own explicit identifier. app.descendants(...)["copyPhotos.section"] resolved unpredictably. The fix was to take the identifier off the container and put one on an unambiguous leaf, the “My Photos” Text.

swipeUp() is a blunt instrument. It drags nearly a full screen height per call, which is fine for a target deep in the page and terrible for a short section near the top: overshoot it and further swipeUp()s can never recover, because they only scroll the same direction. Hence a small-increment variant:

private func smallScrollUntilHittable(_ element: XCUIElement, in scrollView: XCUIElement, maxAttempts: Int = 20) {
    var attempts = 0
    while !element.isHittable, attempts < maxAttempts {
        let start = scrollView.coordinate(withNormalizedOffset: CGVector(dx: 0.5, dy: 0.7))
        let end = scrollView.coordinate(withNormalizedOffset: CGVector(dx: 0.5, dy: 0.55))
        start.press(forDuration: 0.05, thenDragTo: end)
        attempts += 1
    }
}

There’s one more, and it’s the async-rendering trap that every screenshot suite eventually hits. The Insights panel’s container exists the moment its ScrollView renders — before the .task that resolves Market Value has returned. Capturing on the container alone races the fetch and grabs the pre-resolution dash. Wait for the resolved content instead of the container that holds it:

let marketValueResolved = app.descendants(matching: .any)
    .matching(NSPredicate(format: "label BEGINSWITH 'Market Value: $'"))
    .firstMatch
XCTAssertTrue(marketValueResolved.waitForExistence(timeout: Self.longTimeout), "Market Value never resolved")

And the interruption monitor matches English button labels only, in a suite that runs in six languages. That’s deliberate, not an oversight: no permission prompt is expected under --seed-screenshots, so the monitor is a backstop against a genuinely unexpected system alert. A miss means the test hangs and fails, which is what would happen without the monitor anyway.

Nothing Else Matters — Why Not fastlane snapshot

Fair question. snapshot has done this job for a decade, it’s well understood, and frameit handles the device frames. If you’re already on fastlane, keep it.

I didn’t reach for it here for one reason: snapshot captures, it doesn’t verify. Point it at a broken locale configuration and it’ll cheerfully produce ninety-six images, sixteen of them in the wrong language, and hand you a tidy HTML index of your bug. That’s not a knock on the tool — capture is the job it signed up for, and it does it well. But capture was never my problem. Two xcodebuild invocations and a for loop capture fine. My problem was that nothing anywhere in the pipeline ever asked whether the thing that happened was the thing I’d asked for.

The second reason is smaller and more personal. snapshot wants SnapshotHelper.swift vendored into your UI test target and drives locale through its own launch-argument convention. That’s another layer between me and AppleLanguages — and given that a locale mechanism silently not applying is what caused all of this, I wanted that line in my own file, where I could read it and test it.

The valuable part here isn’t the harness. It’s the two guards, and they’re portable: they read PNGs off disk and don’t care what produced them. Run assert_language_hashes_distinct and assert_panels_pairwise_distinct after fastlane snapshot and you get the assertion layer without touching anything else you’ve built.

Fade to Black — What It Actually Cost

One day. The commit log is honest about the shape of it. The first entry is [WIP] capture harness — unverified, no test run completed, and the two that matter most are [FIX] Fail loudly on a repeat of the silent English-panel bug and [FIX] Capture the real My Photos gallery in panel-06, catch cross-panel duplicates.

Neither of those is a translation bug. Both are plumbing bugs — a string, or in the second case an entire screen, that never made it from the source to the pixel. That’s the real category here, and it’s the awkward kind: invisible to eyes, trivial to machines.

It isn’t finished. Three of the eight panels can’t be captured by a simulator at all: the dead wax viewfinder needs a live camera over a real run-out groove, the Home Screen widget isn’t reachable from XCUITest, and CarPlay needs its own simulator. Those are captured by hand and fed to the same compositor (swift Tools/ShotComposer/compose.swift), which doesn’t care about provenance, only that it gets a raw PNG and a caption and emits something opaque, alpha-free, and exactly 1320 × 2868 or 2064 × 2752.

And the captions are still English-only, sitting in a captions.json next to the composer rather than in the app’s String Catalog. That’s the same bug class as the one I opened with, one layer up, waiting: a caption file no translator has ever been shown, in a pipeline that would happily composite English text over a Japanese screenshot and report success. Moving them into Localizable.xcstrings, where extraction is a build-time guarantee and translators already work, is the next thing, and it’ll want a guard of its own. Identical output across locales is a defect until proven otherwise.

The general shape of it: when you build a pipeline that asks some process to do something, make it prove what it did. My original harness sent a locale into a black box and photographed whatever came out, and it would have kept doing that indefinitely. What fixed it wasn’t a better flag — it was shasum -a 256 run twice along two different axes, plus a mode whose only job is to point those guards at known-bad input and fail if they stay quiet.

Ninety-six images, six languages, and not one of them requires me to read Japanese.