What it is
Word error rate, often shortened to WER, is a common way to measure how closely a transcript matches a reference. It is mainly used to evaluate speech recognition: take a transcript that a person has checked and treat it as correct, run the same audio through a recognizer, and compare the two texts word by word. The result is a percentage showing how much the machine’s text differs from the reference.
Because it gives a single number, it is convenient for comparing two tools, two versions of a system or two kinds of audio. That convenience is also why it is often over-trusted.
What it counts
The measure counts three kinds of difference between the reference and the output:
- Substitutions (S): a word was replaced by a different word.
- Deletions (D): a word in the reference is missing from the output.
- Insertions (I): the output contains a word that is not in the reference.
The formula is:
WER = (S + D + I) / N
where N is the number of words in the reference.
A worked example
This example is hypothetical and kept small so the arithmetic is easy to check.
Reference (8 words): “the patient should take two tablets every morning”
Output: “the patient should take to tablets morning”
Machine output like this is typical of a rough first draft. Compare the two word by word:
- “two” became “to”: 1 substitution.
- “every” is missing: 1 deletion.
- Nothing extra was added: 0 insertions.
So S = 1, D = 1, I = 0 and N = 8.
WER = (1 + 1 + 0) / 8 = 2 / 8 = 0.25, or 25 percent.
Notice what this number does not tell you. The output lost the word “every”, which changes how often the tablets should be taken, and it swapped a number for a sound-alike. Both are errors of one word, so they count the same as a trivial slip. In a medical context those are exactly the errors that matter, which is why such material needs checking by a qualified human reviewer.
A second small point: WER can exceed 100 percent. If the output has many insertions, the total of errors can be larger than the number of reference words. For instance, a 4-word reference with an output of 10 unrelated words has 4 substitutions and 6 insertions, so WER = 10 / 4 = 2.5, or 250 percent. This is unusual in practice but shows that the figure is a ratio of errors, not a share of words that are right. That also means that “100 minus WER” is not strictly an accuracy percentage.
How the comparison is made
Words are aligned between the two texts so that the smallest total number of changes explains the difference. Computers do this with a standard alignment method; by hand you can usually see it. The result depends on how the texts are prepared first, which is where much of the confusion arises.
What it leaves out
Meaning
Every word counts equally. A dropped “um” and a dropped “not” cost the same, even though one hardly matters and the other reverses the meaning. Two outputs with the same WER can differ greatly in how misleading they are.
Normalization choices
Before comparing, people usually decide whether to ignore capitalization and punctuation, how to treat numbers (“20” versus “twenty”), contractions, hyphens and spelling variants. Different choices give different scores for the same output, so two reported figures are only comparable if the preparation was the same. Without that information, a quoted figure says little.
Punctuation and formatting
Most WER calculations strip punctuation, so a transcript with no punctuation at all can score well. Yet readability depends heavily on punctuation, casing and paragraphing, and in subtitles on how text is broken into lines.
Timing
WER ignores when words appear. A perfectly accurate transcript that is out of sync with the video would still score zero errors. For subtitles, timing, cue length and reading speed are central to quality.
Speakers and non-speech information
The measure does not check who said what, or whether relevant sounds are noted. Speaker labels and descriptions of important sounds are outside its scope.
Names and rare terms
Because rare words are a small share of the total, a transcript can get most words right and still misspell the key names. A low overall score can hide a cluster of errors on exactly the words a viewer cares most about.
Condensed subtitles
Subtitles often shorten speech on purpose. Compared against a full verbatim transcript, a good subtitle would be penalized for deletions even though the shortening was a sensible editorial decision. WER fits transcripts better than finished subtitles.
The reference itself
The score is only as good as the reference. If the human transcript contains errors or reflects a particular style, the comparison inherits those problems.
How to read a quoted figure
If you see a WER number from a vendor, a paper or a tool, a few questions help:
- What kind of audio was tested: clean studio speech, phone calls, accented speakers, noisy rooms?
- How was the text normalized before comparing?
- How large was the test set, and was it similar to your material?
- Who made the reference, and how was it checked?
- Which errors mattered most for the use?
Results on one kind of audio often do not carry over to another. A tool that does well on clear speech may do much worse with overlapping voices, which is why the factors behind audio quality matter so much. A figure quoted without this context should be treated cautiously, and no figure shows that output is error-free.
Where it is useful
WER is practical for tracking one system on one kind of material over time, comparing candidate tools on a sample of your own audio, or estimating how much editing a draft is likely to need. For that last purpose, it is a rough guide only, since caption editing time depends on which errors there are, not only how many.
Used alongside a manual read for meaning, names and timing, it is a helpful input. On its own, it measures one narrow thing: the proportion of words that differ.