On-device speaker diarization, measured on real meetings

The result, up front

We scored every speaker-diarization pipeline our app ships against 16 real meetings from the AMI corpus, which is 313 minutes of annotated speech, plus a synthetic corpus we built ourselves. Two findings came out of it. The second one is the reason this article exists.

First: for the live, streaming diarizer, the single biggest factor was not the architecture or the configuration. It was which trained checkpoint we loaded. Same model, same audio, same code path, one enum value apart.

We had been shipping the DIHARD 3 checkpoint, because it is what the library API hands you if you do not choose. On meeting audio it is 3.5x worse, and it finds the right number of people in 9 meetings out of 16 where the AMI checkpoint manages 15.

One honest note before anyone tries to reproduce this: the FluidAudio command line tool defaults to the AMI checkpoint, not DIHARD 3. Only the library API defaults to DIHARD 3. If you reproduce this through the CLI you will get the good checkpoint by default and see no gap at all.

The finding that should worry you more

The offline pipeline has a clustering threshold. Here is that one knob swept across both corpora.

Look at the right-hand column. It does not move. Every threshold from 0.60 to 0.95 scores exactly 8.11% on the synthetic corpus, while the same knob spans 6.82% to 32.78% on real meetings, a factor of nearly five.

The synthetic set is not merely optimistic here. It has no opinion whatsoever about a parameter that is worth 5x on real audio. A sweep run against it would have returned a perfectly flat line and we would have concluded, reasonably and completely wrongly, that the threshold did not matter and any value would do.

That corpus is conversations we stitched together from LibriSpeech speakers. The ground truth is exact because we placed every turn ourselves, it costs nothing, and it is deterministic, which is precisely what makes it attractive for sweeping configurations. But stitched read speech has no crosstalk, no room reverberation, and no overlapping turn-taking, and clustering thresholds exist to deal with exactly those things. When speakers never talk over each other, almost any threshold separates them.

We still use it, for regression detection and for iterating quickly. We will not use it to choose a parameter again.

Why we ran this

Lyonesse labels who said what in a recording, entirely on-device. That feature either works or it quietly hands you a transcript where two colleagues are merged into one voice. We had tuned it, which is not the same as having measured it: we had moved thresholds until output looked better on recordings we happened to have.

The library we build on, FluidAudio, publishes its own DER figures. Those are useful but they cannot tell us what our code does with our buffering and our chunk sizes. So we scored our own production code paths, feeding the streaming diarizer in the same real-time chunks the live captions view feeds it, because a correct speaker label that arrives after the speaker has finished is useless in a caption.

Why our best number is optimistic

The checkpoints are not configuration presets. They are separately trained models, each fine-tuned on a different diarization corpus: AMI for meetings, CALLHOME for telephone speech, DIHARD for a deliberately hard mixed-domain set.

So scoring the AMI-trained checkpoint on AMI meetings is an in-domain evaluation. These are AMI's standard evaluation meetings, so it is not testing on training data, but the model has seen the same rooms, the same microphones and the same annotation conventions. 10.87% is an optimistic bound, not a prediction for your audio.

What makes us think the checkpoint is genuinely better rather than merely at home is the synthetic corpus, which is out-of-domain for every checkpoint here. The AMI checkpoint still wins there, 13.93% against DIHARD 3's 24.37%. It wins away from home, so we ship it. We would not defend the 10.87% itself.

Offline beats streaming, but they are not solving the same problem

Our offline pipeline (pyannote segmentation, WeSpeaker embeddings, VBx clustering) scores 6.83% on the same 16 meetings, comfortably ahead of the streaming model's 10.87%. That is the right result and we run it for finished recordings.

It is not a fair fight, though, and it is worth saying so. The offline pipeline sees the entire recording before it commits to anything. The streaming one has to emit a label while the meeting is still happening, and even then its labels finalize 8.8 seconds behind the audio. Those are different jobs, and only one of them can drive live captions.

The latency is also a real cost of the checkpoint we chose. The AMI checkpoint lags 8.8s where DIHARD 3 lags 4.1s. We took the accuracy. The live view paints provisional spans immediately, so captions are still labelled as you watch; what changes is how long a label can still be revised.

DER alone will lie to you

One row in the results deserves singling out. Our online clustering fallback, at a threshold of 0.6, posts a respectable-looking 14.46% DER on the 16 meetings. It also identifies the correct number of speakers in one meeting out of sixteen.

A pipeline that collapses four people into two can still score acceptably, because most of the speech is still attributed to a speaker. To a user it is completely broken. If you are measuring diarization, report the speaker count next to the error rate, because they fail independently.

What this changed in the product

Live speaker labels now run the AMI checkpoint. We also found we had detuned the online clustering fallback well past the library's defaults, which over-split speakers and let short noisy fragments poison the speaker embeddings; that is back to sane values.

The benchmark then caught something we would not have found by reading code. A dependency upgrade landed while we were writing this, and it redefined a constant we had tuned. The offline clustering threshold used to be interpreted as a cosine similarity, cut at sqrt(2 - 2t); it is now the dendrogram cut distance applied directly, so larger values merge more. The value still compiled and still validated. It simply meant something else.

Our first instinct was to convert rather than re-measure: 0.7 became 0.775, the value that preserves the old cut. That was wrong, and the table above shows why. 0.775 scores 7.99% where the untouched 0.7 scores 6.83%, with two fewer meetings counted correctly. The same upstream change had also altered the assignment step, so preserving the cut did not preserve the algorithm. We shipped 0.7 and wrote down the rule: when a library redefines a tuned constant, re-measure it, do not convert it.

Limitations, stated plainly

• This is close-talking headset audio, not far-field. The AMI files we scored are the mixed headset recordings, downmixed to 16 kHz mono. Real meetings with real crosstalk and overlap, but not a single distant microphone. Far-field DER on AMI is typically much worse, so do not compare these figures to published single-distant-mic results.
• Neither corpus is what our users record, which is usually one phone in the middle of a table. Hand-labelling real recordings of that kind is the honest next step and we have not done it.
• In-domain for the winner, as described above.
• We did not reproduce FluidAudio's published figures. Their documented numbers are roughly 12% offline and roughly 30% streaming. We measured 6.83% and 10.87%. Being better on both is not a victory lap, it is a sign that some protocol detail differs: which audio condition, what collar, how overlap is handled, or how the pipeline is fed. Their own documentation now notes that their AMI table predates the threshold change described above. We publish our protocol and data rather than claim a win.
• English only, 16 meetings, one machine.

Methodology, and why you can check it

A benchmark published by a company that sells one of the things being measured deserves suspicion. Here is what ours is made of.

DER is corpus-weighted, and we mean something specific by that. Total error time divided by total reference speech time, across all 16 meetings. Not an average of per-meeting percentages, which would weight a 7-minute meeting the same as a 27-minute one. Our own harness made exactly that mistake until this round, and the two differ by up to 1.6 points here. Note also that "corpus-weighted" is ambiguous in a way that matters: weighting by audio duration rather than by scored reference speech shifts these numbers by a few tenths. Every figure on this page uses reference speech.

Standard scoring conventions. 10ms frames, a 0.25 second collar excluded on both sides of every reference boundary, and overlapped speech excluded from scoring. We exclude overlap because both of our pipelines emit one speaker per instant, so scoring it would measure a capability we have never claimed.

Production code paths. Each pipeline ran through the code our users get, and the streaming diarizer was fed in the same real-time chunks the live captions path feeds it, not handed the whole file at once.

The data is downloadable. Every per-file, per-config measurement, including the absolute error seconds so you can recompute any aggregate you prefer:

• summary.json - every corpus-weighted measurement with per-file breakdowns and the protocol
• raw-results.json - all 432 rows: both corpora, every config, absolute miss/false-alarm/speaker-error seconds
• reference-rttm.tar.gz - the reference annotations we scored against, in RTTM
• corpus-manifest.json - which files, which corpus

The AMI audio is public. We scored all 16 meetings of the EN2002, ES2004, IS1009 and TS3003 sessions, pulled from FluidInference/ami-corpus-mirror. Disagree with our scoring? The RTTMs and per-file numbers are above.

If you are shipping diarization yourself

Check whether your model has a domain-matched checkpoint. This was worth 3.5x and cost one line. We assumed the general-purpose default was the safe choice, and on meeting audio it was nearly the worst option we measured.

Do not tune parameters on clean synthetic audio. Ours was perfectly flat across a knob that moves real-world error by 5x. Use it for regressions; validate on in-domain audio before you commit.

Report the speaker count next to the DER. They fail independently, and the count is what users actually notice.

This is the third benchmark we have published. The first compared Apple's SpeechAnalyzer against Whisper and the second added Parakeet and MOSS. All three changed what we ship.

Related reading

• Offline AI meeting notes
• Private transcription apps compared
• Round 1: Apple SpeechAnalyzer vs Whisper