The result, up front
On LibriSpeech test-clean, people reading audiobooks into a good microphone, Apple's SpeechAnalyzer scores 2.12% word error rate and Parakeet Ultra scores 2.29%. That's close, and Apple is ahead, so for months Apple's engine was the default in Lyonesse wherever it had the language.
Then we ran both on six real meetings from the AMI corpus. That's 3.5 hours of people talking over each other in a room, about 33,600 words. The order flips:
Parakeet makes 18% fewer errors, and it wins all six meetings. Even the live captions, which only see a few seconds of audio at a time, beat Apple's pass over the whole file on every one of them.
| Meeting | Apple SpeechAnalyzer | Parakeet Ultra | Parakeet Ultra, live |
|---|---|---|---|
| ES2004b (39 min) | 18.7% | 15.9% | 16.5% |
| ES2004d (37 min) | 25.6% | 21.8% | 22.7% |
| IS1009b (34 min) | 19.5% | 16.2% | 16.1% |
| IS1009c (30 min) | 16.2% | 12.6% | 13.0% |
| TS3003b (37 min) | 18.7% | 14.6% | 14.6% |
| TS3003d (44 min) | 25.2% | 19.9% | 20.4% |
| All six | 20.9% | 17.1% | 17.5% |
Word error rate, lower is better. AMI Meeting Corpus, mixed headset audio at 16 kHz, scored against the official manual transcripts, which keep every "um" and false start, so the numbers are high for every engine. Apple ran through SpeechAnalyzer with SpeechTranscriber at en_US. Parakeet Ultra ran through FluidAudio 0.18.3. The live column uses the 6-bit encoder described below, the one the app will use. Live means FluidAudio's sliding window with 8 s of left context, a 5 s chunk and 2 s of right context. M2 Pro, macOS 27.
Why the clean benchmark pointed the wrong way
LibriSpeech is one person reading prose they've practised, with no crosstalk. A meeting is several people interrupting each other, trailing off and correcting themselves. A model can be excellent at the first and merely good at the second, and a 0.17 point lead on audiobooks says nothing about which one you're getting.
We had the right data the whole time, for a different job. We used these AMI meetings in Round 3 to pick a diarization model, and never pointed a speech recognizer at them. If your users record conversations, benchmark on conversations. It sounds obvious, and we still missed it for months.
Making the model smaller without making it worse
Parakeet has one real cost next to Apple's engine: you have to download it. The version FluidAudio ships is about 600 MB, and almost all of that is the encoder, 568 MiB of 8-bit weights. So we re-exported the encoder from the original weights and tried a few ways of compressing it with Core ML's palettization, which stores each weight as an index into a small table of values.
| Encoder | Size | LibriSpeech (300 clips) | Meeting-style set | AMI |
|---|---|---|---|---|
| 8-bit, as FluidAudio ships it | 568 MiB | 1.67% | 9.25% | 17.11% |
| 6-bit palette, groups of 16 | 440 MiB | 1.71% | 9.20% | 17.08% |
| Mixed 4-bit and 8-bit | 367 MiB | 1.75% | 10.97% | not run |
| 4-bit palette, groups of 16 | 284 MiB | 1.78% | 11.44% | not run |
The meeting-style set is our own: 9 scripted conversations in German, English, French and Dutch, read by synthetic voices with room and phone noise added. It isn't real speech, so we use it to rank options and AMI to confirm them. Its numbers aren't comparable with the AMI column.
The 6-bit encoder is 128 MiB smaller and scores the same as the original everywhere we looked, including live captions on the meeting-style set (9.88% vs 9.86%). The difference is smaller than run-to-run noise.
The 4-bit one is the warning. On LibriSpeech it looks fine, 0.1 points worse for half the size. On meeting-style audio it's 2.2 points worse. That's the same trap as above, a second time: the clean benchmark would have happily shipped the model that's worse at the one thing our users do. We also tried 4-bit linear quantization, which never finished loading on the Neural Engine, and an 8-bit palette, which came out bigger than what we started with.
With the decoder and vocabulary added, the whole model is now 470 MB instead of about 600 MB.
What we changed in Lyonesse
Parakeet Ultra is now the default engine for the 25 European languages it covers, starting with version 2.0.18. It downloads in the background when you install or update the app, through Apple's Background Assets, so nobody has to find a download button in Settings first. It waits for Wi-Fi and stays off cellular and Low Data Mode.
Apple's engine stays as the fallback. It covers Japanese, Korean and Chinese, which Parakeet doesn't, and it handles recordings made before the model has arrived. You can still pick either engine in Settings.
The trade-off is latency. Parakeet's live captions show up about 7 seconds behind the speaker, because each window needs a bit of audio after the words it confirms. Apple's captions are close to instant. So the recorder now says, during those first seconds, that recording is working and the first words are on their way. We think a more accurate transcript is worth a 7 second wait, but it's a real cost and we'd rather say it out loud than have people think the app is stuck.
What this doesn't show
- The real-meeting numbers are English only. AMI has no other languages. Our meeting-style set covers German and French too, where Parakeet also wins, but that's synthetic speech.
- AMI's headset mix is cleaner than a phone on a table in the middle of the room. Both engines will do worse there, and we haven't measured whether the gap grows or shrinks.
- One machine, an M2 Pro. Earlier rounds found Apple's output identical across chips, but we haven't yet run the 6-bit encoder on an iPhone's Neural Engine.
Reproduce it
- summary.json: every number in this post, per meeting, with the setup
- Every transcript, per meeting: Apple SpeechAnalyzer, Parakeet Ultra 8-bit, Parakeet Ultra 6-bit, Parakeet Ultra 6-bit, live
- The audio and reference transcripts are the AMI Meeting Corpus (CC BY 4.0), meetings ES2004b, ES2004d, IS1009b, IS1009c, TS3003b and TS3003d.
Related reading
- Round 1: Apple SpeechAnalyzer vs Whisper, measured
- Round 2: Parakeet and MOSS on the same corpus
- Round 3: on-device speaker diarization on real meetings
- Apple said macOS 27 improves speech accuracy. We measured the API.
Get Lyonesse
Private, on-device transcription with AI summaries and cross-library Q&A. Works on iPhone, iPad, and Mac.
Get Lyonesse on the App Store