Apple said macOS 27 improves speech accuracy. We measured the API.

The result, up front

With iOS 27 and macOS 27, Apple says dictation "captures what users say as polished text with greater precision" and promises "a major boost in accuracy" from its most advanced on-device model yet. So we re-ran the corpus from our Apple-versus-Whisper benchmark on macOS 27 and compared the output, transcript by transcript, against the transcripts we published from macOS 26.5.1.

For the API that apps actually call, nothing moved. Of 300 LibriSpeech utterances, 280 came back byte-identical to the macOS 26.5.1 run on the same Mac. 296 of 300 were identical once punctuation is ignored. Four transcripts differ in any word at all, and two of those are the recognizer writing "20" where it previously wrote "twenty", and "second" where it previously wrote "2nd" — formatting flipping in both directions, not accuracy moving in one.

We also ran it on a second machine, an M4 Pro, because Apple's advanced on-device model requires an M3 or later Mac with at least 12GB of memory. The two machines returned the same words on all 300 clips.

The M2 Pro and M4 Pro runs agree with each other on 293 of 300 transcripts byte-for-byte, and on all 300 once punctuation is ignored. The seven differences are commas. Switching system dictation on and repeating the M4 Pro run changed nothing: 300 of 300 byte-identical to the same machine's earlier run.

The finding we did not expect

Apple's Speech framework ships two transcription modules, and on this corpus they are 4.3x apart.

Same framework, same analyzer, same audio, same machine, one module swapped. DictationTranscriber matched the SpeechTranscriber output byte-for-byte on exactly one clip in 300. It is a different model, and on read speech it is a much weaker one:

This is worth knowing before you pick a module by its name. If you are building a dictation feature, "DictationTranscriber" is the obvious-looking choice, and on this corpus it is the wrong one by a factor of four. The caveat runs the other way too: LibriSpeech is read prose, not someone dictating a message to their phone, and the two modules are built for different jobs — DictationTranscriber exists for short, interactive input and covers locales SpeechTranscriber does not. We are not claiming it is a bad module. We are saying that if your audio looks like transcription rather than dictation, measure before you choose, because the gap is not small.

Why Apple's claim and this measurement do not contradict each other

Apple's accuracy promise is about system-wide dictation — the microphone key on the keyboard — and it is tied to a specific on-device model that runs only on an M3 or later Mac with at least 12GB of unified memory (and on iPhone 17 Pro, iPhone Air and later). Nothing in that announcement says the Speech framework gets the same model, and our measurement is that it returns what it returned in macOS 26.

The SDK agrees. Diffing the Speech framework's Swift interface between the macOS 26 and macOS 27 SDKs, everything added in 27 is audio plumbing:

• AnalyzerInputConverter — converts buffers to the analyzer's format for you
• AssetInputSequenceProvider — feeds an AVAsset straight in
• CaptureInputSequenceProvider — feeds an AVCaptureDevice straight in
• AnalyzerInput gains CMReadySampleBuffer input plus bufferDuration and bufferFormat, and its buffer property is deprecated
• SpeechAnalyzer.Options.ignoresResourceLimits becomes public
• SFSpeechError.Code.cannotConfigureAudioSystem is added

No new transcriber, no new preset, no new content hint, no new model selector. Useful additions if you have been hand-rolling an AVAudioConverter, and none of them touch accuracy.

Why we compared strings instead of just scoring

A word error rate alone cannot answer "did the model change". Two different models can land on the same WER, and a 0.03 percentage point move on a 300-clip subset tells you nothing either way.

Byte-identical transcript counts can answer it. A changed model does not reproduce the same string, with the same commas, on 280 clips out of 300; an unchanged one produces almost nothing else. That is what makes a null result here worth publishing rather than merely inconclusive — and it is only possible because the Round 1 transcripts were published in full, per utterance, fifteen months before we needed them for a comparison nobody had planned.

If you ship anything that depends on a system model, publishing your raw outputs is how you find out later whether the vendor changed it under you.

Reproduce it

The whole thing is a 40-line Swift file, no Xcode project and no test host, and it runs in about a minute per 300 clips:

• run.swift - the runner; swiftc -O run.swift -o run
• compare.py - fetches the published macOS 26.5.1 transcripts and diffs a run against them
• summary.json - every measurement, the full protocol, both machines' configuration

Every per-utterance transcript from all five runs: SpeechTranscriber M2 Pro, SpeechTranscriber M4 Pro, SpeechTranscriber M4 Pro with dictation on, DictationTranscriber M2 Pro, DictationTranscriber M4 Pro. The macOS 26.5.1 baseline is the Round 1 transcript package. Disagree with the normalizer? Rescore all of it.

What we changed in our own product

Nothing, and that is the useful part of a null result. We publish these benchmarks because they decide what Lyonesse ships: Round 1 moved our default engine to Apple's new API, Round 2 retired Whisper in favour of Parakeet, and Round 3 changed which diarization checkpoint we load. This time the measurement said the ground had not moved, so we left our defaults alone instead of shipping a change we could not justify.

If you are on iOS 27 or macOS 27 and your users report better dictation, they are probably right — that is the keyboard, and we have not measured it. If you are a developer hoping the framework quietly got better, it did not, on either machine we could test.

Related reading

• Round 1: Apple SpeechAnalyzer vs Whisper, measured
• Round 2: Parakeet and MOSS on the same corpus
• Round 3: on-device speaker diarization on real meetings

Frequently Asked Questions