Apple's speech API wins on audiobooks. On real meetings, Parakeet wins.

On clean read speech Apple is slightly ahead. On 3.5 hours of real meetings Parakeet Ultra makes 18% fewer errors and wins every meeting. So it's now our default, and we got the model 22% smaller without losing a word.

Apple's speech API wins on audiobooks. On real meetings, Parakeet wins.

The result, up front

On LibriSpeech test-clean, people reading audiobooks into a good microphone, Apple's SpeechAnalyzer scores 2.12% word error rate and Parakeet Ultra scores 2.29%. That's close, and Apple is ahead, so for months Apple's engine was the default in Lyonesse wherever it had the language.

Then we ran both on six real meetings from the AMI corpus. That's 3.5 hours of people talking over each other in a room, about 33,600 words. The order flips:

Parakeet Ultra, whole file17.1%
Parakeet Ultra, live (7 s lag)17.5%
Apple SpeechAnalyzer, whole file20.9%

Parakeet makes 18% fewer errors, and it wins all six meetings. Even the live captions, which only see a few seconds of audio at a time, beat Apple's pass over the whole file on every one of them.

MeetingApple SpeechAnalyzerParakeet UltraParakeet Ultra, live
ES2004b (39 min)18.7%15.9%16.5%
ES2004d (37 min)25.6%21.8%22.7%
IS1009b (34 min)19.5%16.2%16.1%
IS1009c (30 min)16.2%12.6%13.0%
TS3003b (37 min)18.7%14.6%14.6%
TS3003d (44 min)25.2%19.9%20.4%
All six20.9%17.1%17.5%

Word error rate, lower is better. AMI Meeting Corpus, mixed headset audio at 16 kHz, scored against the official manual transcripts, which keep every "um" and false start, so the numbers are high for every engine. Apple ran through SpeechAnalyzer with SpeechTranscriber at en_US. Parakeet Ultra ran through FluidAudio 0.18.3. The live column uses the 6-bit encoder described below, the one the app will use. Live means FluidAudio's sliding window with 8 s of left context, a 5 s chunk and 2 s of right context. M2 Pro, macOS 27.

Why the clean benchmark pointed the wrong way

LibriSpeech is one person reading prose they've practised, with no crosstalk. A meeting is several people interrupting each other, trailing off and correcting themselves. A model can be excellent at the first and merely good at the second, and a 0.17 point lead on audiobooks says nothing about which one you're getting.

We had the right data the whole time, for a different job. We used these AMI meetings in Round 3 to pick a diarization model, and never pointed a speech recognizer at them. If your users record conversations, benchmark on conversations. It sounds obvious, and we still missed it for months.

Making the model smaller without making it worse

Parakeet has one real cost next to Apple's engine: you have to download it. The version FluidAudio ships is about 600 MB, and almost all of that is the encoder, 568 MiB of 8-bit weights. So we re-exported the encoder from the original weights and tried a few ways of compressing it with Core ML's palettization, which stores each weight as an index into a small table of values.

EncoderSizeLibriSpeech (300 clips)Meeting-style setAMI
8-bit, as FluidAudio ships it568 MiB1.67%9.25%17.11%
6-bit palette, groups of 16440 MiB1.71%9.20%17.08%
Mixed 4-bit and 8-bit367 MiB1.75%10.97%not run
4-bit palette, groups of 16284 MiB1.78%11.44%not run

The meeting-style set is our own: 9 scripted conversations in German, English, French and Dutch, read by synthetic voices with room and phone noise added. It isn't real speech, so we use it to rank options and AMI to confirm them. Its numbers aren't comparable with the AMI column.

The 6-bit encoder is 128 MiB smaller and scores the same as the original everywhere we looked, including live captions on the meeting-style set (9.88% vs 9.86%). The difference is smaller than run-to-run noise.

The 4-bit one is the warning. On LibriSpeech it looks fine, 0.1 points worse for half the size. On meeting-style audio it's 2.2 points worse. That's the same trap as above, a second time: the clean benchmark would have happily shipped the model that's worse at the one thing our users do. We also tried 4-bit linear quantization, which never finished loading on the Neural Engine, and an 8-bit palette, which came out bigger than what we started with.

With the decoder and vocabulary added, the whole model is now 470 MB instead of about 600 MB.

What we changed in Lyonesse

Parakeet Ultra is now the default engine for the 25 European languages it covers, starting with version 2.0.18. It downloads in the background when you install or update the app, through Apple's Background Assets, so nobody has to find a download button in Settings first. It waits for Wi-Fi and stays off cellular and Low Data Mode.

Apple's engine stays as the fallback. It covers Japanese, Korean and Chinese, which Parakeet doesn't, and it handles recordings made before the model has arrived. You can still pick either engine in Settings.

The trade-off is latency. Parakeet's live captions show up about 7 seconds behind the speaker, because each window needs a bit of audio after the words it confirms. Apple's captions are close to instant. So the recorder now says, during those first seconds, that recording is working and the first words are on their way. We think a more accurate transcript is worth a 7 second wait, but it's a real cost and we'd rather say it out loud than have people think the app is stuck.

What this doesn't show

  • The real-meeting numbers are English only. AMI has no other languages. Our meeting-style set covers German and French too, where Parakeet also wins, but that's synthetic speech.
  • AMI's headset mix is cleaner than a phone on a table in the middle of the room. Both engines will do worse there, and we haven't measured whether the gap grows or shrinks.
  • One machine, an M2 Pro. Earlier rounds found Apple's output identical across chips, but we haven't yet run the 6-bit encoder on an iPhone's Neural Engine.

Reproduce it

Related reading

Get Lyonesse

Private, on-device transcription with AI summaries and cross-library Q&A. Works on iPhone, iPad, and Mac.

Get Lyonesse on the App Store