Possibilities, not forecasts
Nobody can say for certain how subtitle technology will develop. Tools change quickly, claims in marketing are often ahead of real performance, and some ideas that sound promising never become practical. What follows is a set of directions that researchers and developers are generally working on. Each is a possibility, and each comes with limits.
Better speech recognition
Speech recognition has improved over the years, and further gains are plausible. Areas where progress would be useful include:
- Noisy and overlapping speech. Recordings with crosstalk, music or distance from the microphone remain hard. Better handling of audio quality problems would help many everyday videos.
- Accents and dialects. Performance has historically been uneven across speakers. Wider and more balanced training material could narrow gaps, though it is difficult to know how far. The problem of accent recognition is likely to remain a concern for some time.
- Rare words and names. Tools that adapt quickly to a custom vocabulary or to the context of a video could do better on names and technical terms.
- Several languages in one recording. People often switch languages mid-sentence, and systems that cope with this gracefully would be valuable.
Even if averages improve, individual errors will not disappear, which is why measures such as word error rate will probably continue to be used with caution.
Processing on the user’s own device
At present, many captioning tools send audio to remote servers. A possible direction is running models directly on a phone, laptop or television. The potential advantages include:
- Privacy. If audio never leaves the device, there is less to worry about in terms of caption privacy, which could matter for personal or confidential recordings.
- Lower delay. Skipping the round trip to a server can reduce lag, which is especially important for live use.
- Availability without a connection. Captions might work on a flight or in a place with poor coverage.
There are trade-offs. Smaller models that fit on a device may be less accurate than the largest ones, and running them can use battery and processing power. Whether the balance is acceptable will probably vary by device and by task.
Smarter use of context
Present-day tools often treat each short chunk of speech in isolation. A possible improvement is using wider context: the topic of the whole video, who is speaking, earlier mentions of a name, or even what is visible on screen. This might help with ambiguous words, consistent terminology and the choice between formal and informal address in translation. Using images alongside sound is a research area, and results so far should be treated with caution.
Better translation of spoken material
Machine translation may become more sensitive to tone, register and cultural background. Possible gains include more natural condensing of long sentences to fit reading speed, more consistent handling of recurring terms through shared memory of approved translations, and fewer chained errors when speech recognition and translation work as one process rather than two. Humor, wordplay and idioms are likely to stay difficult, and cultural judgments will probably still need human attention.
Live captioning with less delay
For live events and meetings, automatic captions can be delivered with short delay, but accuracy and speed pull against each other. Systems may keep improving at revising text as more context arrives, although that creates a different problem: text that changes on screen can be tiring to read. Methods for showing corrections without distracting the viewer are an open question.
More personal presentation
Technology may allow subtitles to adapt to the viewer, with their own choices on size, color, position and language. Related ideas include reading-level adjustments or summaries for people who find fast text difficult. These raise questions of their own, such as who decides how much a subtitle may be changed from what was said. Viewer-controlled display is covered elsewhere on the site.
Accessibility beyond speech
Captions for deaf and hard of hearing viewers contain more than words: they identify speakers and describe significant sounds. Automatic detection of non-speech information is an area that remains less developed than transcription, and progress there would be of real practical benefit. Likewise, subtitles in immersive environments such as virtual reality raise new questions about where text should sit in a three-dimensional space.
Things that are unlikely to change quickly
- The need for review. As long as errors occur, someone should check material that matters. For medical, legal or safety content this is expected to remain true.
- Regulation and standards. Accessibility requirements differ by place and will keep changing. Organizations should follow guidance from the relevant authority.
- Questions of fairness. If tools perform better for some languages and accents than others, access will be uneven. This is worth monitoring, whatever the average figures say.
- Costs and business decisions. Which tools become cheap or free depends on business models that nobody can predict.
How to think about new tools
When a new subtitling product or feature appears, a few questions help: How was it tested, and on what kind of audio? Which languages and accents does it handle? Where does the data go? What happens when it is wrong, and how easily can a person fix it? Answers matter more than headline claims. The most useful developments are likely to be those that make careful human review faster, not those that promise to remove it.