One WER number hides everything

My audio eval scorecard has an errors column. Across sixteen runs, it read zero. Sixteen times.

The product was broken. One of the five languages I support produced an empty board: the speaker talked, the microphone was live, the run completed cleanly, and nothing appeared on the canvas.

Both of those things were true at the same time, and the gap between them is the whole post.

I have the rule written at the top of my eval doc now. One WER number hides everything that matters. Not because word error rate is a bad metric, but because a single figure averaged over languages, providers, noise levels and microphones will keep telling you the pipeline is healthy right up until a user opens the app in the language you tested least.

The thing being measured

The app is a voice whiteboard. You talk, it listens, and a diagram builds itself on a canvas you can still drag around afterwards. It supports five languages: English, Malayalam, Arabic, Hindi and Tamil. There are two speech providers behind a switch, which I will call provider A and provider B, because which vendor is which is the least interesting part of this.

The important structural fact is that speech recognition is not the product. It is one link in a chain: microphone, then transcription, then a cleanup pass, then a planner that decides what shapes and connections the scene should contain, then a renderer that puts them on the board.

Cascade errors are multiplicative. A word error rate measures one link of that chain and then gets quoted as if it were the whole thing. That is how you end up optimising the one stage that was already fine.

Stratify, never aggregate

The first principle in the strategy doc is that every scorecard is a matrix of metric by condition, with per-slice counts visible. No scalar summary at the top. If a reader wants the headline they can look at the worst cell.

The stratification axes are language, code-switch density, noise tier, and microphone tier. Noise tier means clean, roughly 15 dB, and roughly 5 dB by synthetic mixing, because production reality lives somewhere between 2 and 14 dB SNR and a clean recording tells you nothing about that.

Sample size per edge condition is 100 to 200 utterances. Below about 50, a winner flips on noise alone, which I found out the boring way.

The corpus is 30 to 60 real recorded sessions replayed through the harness. Real spontaneous speech only. Spontaneous production speech scores worse than clean read-speech benchmarks by a multiple, not a margin, so read audio does not predict anything you care about.

The metrics, and why WER is not the headline

Word error rate is in the table, but only as a trend line, and only after normalising casing, punctuation and numerals. The metrics that actually decide things are these.

CER is the primary metric for Malayalam and Tamil. Both are agglutinative, so one wrong suffix is not one word error, and WER punishes a nearly-correct output as harshly as a wrong one.

MER covers code-switched segments: character error rate on the Indic-script spans, word error rate on the Latin spans, because my users switch scripts mid-sentence constantly.

A meaning-equivalence judge score handles the cases where a string comparison is simply wrong: colloquial against formal register, a word written in Latin script against the same word in Devanagari, different numeral formats, valid suffix variants. Temperature 0, pinned model version, versioned prompt, two or three few-shot examples per language.

Entity preservation is the percentage of named entities, numbers and domain terms that survive. Board labels are mostly entities, so this predicts downstream quality better than WER does.

Hallucination rate is measured on a non-speech set: silence, music, ambient sound. The metric is the percentage of non-speech clips that produce any text at all, scored as a per-utterance binary, because insertion error rate is undefined when the reference is empty.

Script-mismatch rate is specific to this product: output in a script outside the user's declared language set. It sounds like a nitpick. It is not, and I will come back to it.

The judge is a model, and a model's scoring shifts when its version changes. So there is a 50-sample calibration set that gets rescored whenever the judge model changes, and anything over 1 point of drift gets flagged before the numbers are trusted.

Pairing, and error bars that mean something

Every comparison runs the same fixtures through both candidates and reports the paired delta with a confidence interval. Never two separate averages. Pairing cuts the required sample size by a factor of 2 to 5, which is the difference between an eval a solo builder can run and one they cannot.

The bootstrap resamples whole sessions, not single utterances, because utterances inside one session share a speaker, a microphone and a room. Treat them as independent and the per-utterance standard errors understate the real error by more than 3x. That understatement is exactly the kind of thing that makes you confident about a change that does nothing.

At least 1000 resamples, 95% interval from the percentiles. The rule that makes it binding: a provider, prompt or model switch requires the interval to exclude zero. If it does not, I am shipping noise and calling it a decision.

There is also a sealed holdout, 5 to 10 percent, never looked at during development and scored only at release points. And a fixed regression set that runs on every change, with about 20 percent of the rolling set rotated on each major model or prompt change.

Three modes, so the blame lands in the right place

This is the part I would keep if I had to throw the rest away. Every end-to-end fixture runs three ways.

1. Audio. The full pipeline. This is the product number.
2. Golden transcript fed into cleanup and the planner. The delta against mode 1 is the loss attributable to speech recognition.
3. Golden scene plan fed into the renderer. The delta between mode 2 and mode 3 is the loss attributable to the planner.

The conclusion that falls out of this matrix is the reason it exists: if 70 percent of end-to-end loss is downstream of transcription, then provider WER deltas below that floor do not matter. You can win the speech shootout decisively and ship a product nobody can use.

Without the three modes I would have spent the whole budget on the provider switch, because that is the number that has a leaderboard attached to it.

Latency and the soak ladder

Quality is not the only axis. A whiteboard that draws the right diagram four seconds late is a different product from one that draws it instantly, so there are p95 budgets, treated as release gates rather than aspirations.

And an evidence ladder of three soak runs: 5 minutes as a smoke test with pauses, a correction, a digression and a return; 20 minutes of mixed short and long turns for endurance; 60 minutes of continuous representative material as the single-user gate.

The soak pass conditions are mechanism checks, not score checks. Zero unresolved items after the microphone stops and the scene drains. A failed transcription must not block later completed turns. Empty and failed transcripts counted explicitly rather than silently disappearing. No final transcript merged with the next provider turn. The board reaches a ready state after the last visual operation finishes.

What I keep from a soak run is counts and timings, never transcript text. That is deliberate. My own multilingual speech is personal data, and an eval harness is not a good reason to leak it.

The table that broke the story

Here is the run that started this post. Eight fixtures, both providers, sixteen runs, from a single afternoon.

Errors: zero, in all sixteen.

Average shapes per language: 13.8 for provider A, 25.0 for provider B. Read that pair of numbers on their own and provider B wins comfortably, which is exactly what I wanted to conclude, because provider B also led on turn count in four of the five languages.

What the aggregate hid

Malayalam produced zero shapes on both providers. An empty board, from the language I speak at home, hidden inside a healthy-looking average of 25.

Hindi produced zero shapes on provider A despite two returned turns. So provider A does not fail uniformly. It fails per language, at a stage the provider comparison was never designed to look at.

The turn counts are not even comparable across providers, which took me embarrassingly long to see. Provider A returns fewer, longer segments; provider B chops the same audio into many short ones. Nineteen turns against two is a difference in segmentation policy, not in accuracy. If I had ranked providers on turn count I would have been ranking their endpointing behaviour and calling it transcription quality.

The chimes-only fixture is the one that made me add the script-mismatch metric. It is a non-speech clip. Provider A returned nothing, correctly. Provider B returned one line of text in a language nobody had spoken. In the English fixture the same provider dropped two short fragments in two other scripts into the middle of an English talk.

Every one of those failures is invisible to an aggregate error rate. A hallucinated line on a silent clip has no reference to be scored against. A wrong-script fragment inside a correct English transcript barely moves the average. But on the board, they each become a shape with confident-looking text on it, in front of a user who never said that.

This is also why the failure list in my eval doc is mined from real sessions rather than happy paths, and why each row names a root cause rather than just a symptom. Duplicate finals, 82 of them for roughly 41 spoken turns, doubling both traffic and billing. A session that collapsed to 4 visible shapes out of 53 because an autonomous archive treated a single-topic session as a topic change. Twelve accepted connections between concepts that never materialised as a single arrow on the board.

None of those show up in a transcription metric. All of them are what the user experiences as the product being broken.

There are two rules I now treat as binding. A fix may not edit the eval that measures it in the same change, except to add cases. And every eval asserts on the mechanism, the trace events and operation types and entity checks, not only on an outcome score, because a system that learns to please the judge still fails the mechanism check.

The takeaway

Evaluate by subsystem, or you will optimise the wrong one. That is the entire lesson, and the three-mode ablation is the cheapest way I know to buy it.

The rigour here reads like a team's worth of work, and it is not. A solo builder can get most of it in a weekend.

Record eight fixtures that cover your real conditions, including one deliberately silent or non-speech clip. Replay them through both candidates and write the per-slice counts into a table with no summary row. Add the golden-transcript mode so you can see how much of your loss is even attributable to the stage you were about to spend a month on. Pull p95 latencies from a diagnostics dump. Run one 5-minute soak and check that nothing is left unresolved after you stop the microphone.

That is a weekend. What it buys you is the ability to say which stage is broken, in which language, instead of watching a green scorecard while the board stays empty.

The number I trusted said zero errors. The table said Malayalam produced nothing at all. The table was right.