A different kind of screen
On a television or phone, subtitles sit in a flat rectangle, usually near the bottom. In virtual reality (VR) and augmented reality (AR), there is no fixed rectangle. The viewer can turn their head, move through space and look at things in any direction, and the content may fill the whole view. This makes the basic question of where to put the words much harder.
Immersive subtitles have to serve the same people as any other subtitles, including deaf and hard of hearing users, people in a second language and anyone who finds speech difficult to follow. The technology is changing quickly, and practice is still developing, so the points here are general principles rather than settled rules.
The placement problem
A flat-screen caption can stay in one place because the picture stays in one place. In an immersive space, there are several options, each with drawbacks.
- Fixed to the viewer’s head. Text stays at the same spot in the field of view as the head turns. It is always visible, but it can feel like a sticker on the eyes, can block objects, and may cause discomfort if it is too close.
- Fixed in the world. Text sits at a place in the scene, such as a wall. It feels natural but can be missed if the viewer looks the other way.
- Following with lag. Text drifts into view when the viewer turns, moving smoothly after the head. This can be easier on the eyes than rigid locking, but if it lags too much, it may be hard to find.
- Attached to the speaker. Text appears near the person or object that is talking. It helps show who is speaking and where sound comes from, but moving speakers and speakers outside the view cause trouble.
Many experiences let the user choose among such modes, which helps because comfort varies from person to person.
Depth and distance
In a 3D display, text also has a distance from the eyes. If captions appear at a very different depth from the thing the viewer is looking at, the eyes must keep refocusing, which can be tiring. Placing text at a comfortable distance, often similar to the main action, is generally better than putting it very close to the face. If text is drawn in front of objects that are closer than the text appears to be, the result can look wrong and add to discomfort.
Hardware also differs. Resolution, field of view and optics vary between devices, so a size that is readable on one headset may be fuzzy on another.
Comfort and motion
Some people are sensitive to motion in immersive environments, and moving text can add to that. Sudden jumps in the position of text, rapid movement or flickering may be uncomfortable. Smooth, predictable behavior and the option to reduce movement are sensible choices. Designers should also avoid placing text at the extreme edges of the view, where it is hard to see without turning the head or straining.
Because the effects vary between individuals, it is wise to test with a variety of users and allow people to adjust settings. This content is general, and nobody should assume that a particular approach will suit everyone.
Readability
The usual guidance for text applies, with extra care for the environment.
- Size. Text should be large enough to read at the display’s resolution, and users should be able to enlarge it.
- Contrast. The scene behind the text can change constantly, from bright sky to dark rooms. A background box or outline can help the words stay legible. The ideas in font, size and contrast for subtitles carry over.
- Line length and duration. Short lines and sufficient time on screen are still important, and reading speed is the same human limit it is anywhere else.
- Clutter. Immersive scenes may already contain menus, labels and interface elements, and captions need room without covering them.
Directional and spatial sound
Immersive audio often comes from specific places. A footstep behind the viewer or a voice to the left may carry information about where to look. For someone who cannot hear it, that spatial cue is lost unless the interface provides it another way. Options include:
- an arrow or marker pointing toward the source of speech or an important sound
- a label that names the speaker
- a short description of off-screen sounds, as in other non-speech cues
- visual or haptic signals for important events
Such cues can add clutter, so they are best kept for sounds that matter.
Augmented reality in daily life
In AR, text may be overlaid on the real world through glasses or a phone. The same placement questions apply, with extra safety and context concerns. Text that blocks the view of a street, or that is too small to read outdoors in bright light, can be a problem. Live captioning of conversations through wearable displays is an area of active development, and accuracy, delay and privacy are open issues that users should weigh. Automated text may contain mistakes, so it should not be relied on for critical or medical information without checking.
Designing for choice
Since there is no single best answer, the practical lesson is to offer options.
- Provide captions from the start, rather than as a late patch.
- Let users pick the placement mode, size, color and background.
- Offer cues for the direction of speech or sound.
- Avoid requiring fast reactions to text alone.
- Test with people who use captions, including in the actual hardware.
Rules about accessibility in immersive media are still developing and vary by place. Anyone building for a regulated setting should check with the relevant authority or a qualified professional.