It's easy to focus on the visual when you're developing a virtual reality or augmented reality experience. If the images don't look just right, visitors step out of your world and back into the one they know.
But sound can also play a role in the best VR and AR experiences. And the technology you need to create those immersive soundscapes is more accessible than ever before.
How Does Sound Improve Virtual Reality?
The "v" in VR doesn't stand for "visual." But it could. Virtual reality drops people into fantastic landscapes where the unthinkable could happen. Proper sound can deepen the experience and make it seem even more real.
Our ears are capable of localization, says Timothy Duncan, professor of audio and music technology at Cogswell Polytechnical College. We use our ears to tell us where the source of the sound is located, and we can use that sound to determine whether something is stationary or moving. If the sound and the images mesh, the result is an immersive experience.
"Visual elements tell the story, but sound is just as important in virtual reality because it affirms whether or not the experience is believable, and it creates emotional impact," he writes.
Addressing Motion with Sound
If images and noises must mesh to be believed, the sounds should shift as an object or the viewer moves. It's critical, and in the past, it wasn't easy to do.
Stereo headphones do a good job of separating sounds, says Mathias Johansson, CEO and cofounder of audio technology company Dirac Research. If you're listening to a game, you might hear the announcer in one ear and the coaches in the other. But those positions are fixed, even if you move your head from side to side.
"The most basic problem is head motion: When we turn our heads in VR, the soundscape needs to change to reflect the corresponding change in distance or direction. If it doesn’t, then the reality of the virtual experience isn’t so real," explains the team at tech market intelligence platform CB Insights.
Binaural recording systems use microphones placed on both sides of a model's head. They capture noises as they might hit your ears if you were standing in the same spot. While binaural audio isn’t new, the delivery with headphones can be transformative, especially when tied into audio software programming.
Consider work done at the University of Rochester. The team hopes to give people an immersive experience inside an orchestra. You'll not only hear the sound, but you'll walk through the pit and hear the sounds change with each step and movement you make.
“Previous AR/VR research has focused mainly on the visual side. You see the 3-D, but you don’t really hear the audio change when you move your head or look at different objects. The purpose of this proposal is adding 3-D sound. That’s what’s novel here," says Zhiyao Duan, assistant professor in the Department of Electrical and Computer Engineering at the University of Rochester.
Sound Signals Are Unique
While dummies can help engineers capture multiple sounds for one idealized person, it’s not enough. To find truly immersive sound, researchers need to do a little more.
"Everyone's ears are unique, so the imprint of one person's anatomy on a sound is completely different from the other. It's the reason generic dummy head binaural recordings don't have the same effect on everyone. Likewise, they don't always work for VR either," explains Mona Lalwani, executive producer at Vox.
Individualized recordings that account for a person's anatomy would be the answer; however, creating a soundscape like this means taking multiple measurements. "The subject’s head and ear shape, as well as sounds bouncing off of the shoulders, all contribute to a fairly accurate picture of the full frequency spectrum," Steve Horowitz and Scott Looney write at Sonic Scoop.
Gains in recording technology and computer modeling could make this sound concept more accessible. If you have access to a bank of data gleaned from hundreds of individuals, you could make predictions about the sounds one person should hear.
How HQ Virtual Reality Sound Differs From Standard Sound
Most entertainment projects involve an element of sound. Ever since talkies were introduced, music and dialogue have become an important part of how we consume stories. But traditional soundscapes can interfere with a truly immersive experience.
A traditional, swelling soundtrack can be confusing in a VR environment, write Ustwo Games’ Todd Baker and Manesh Mistry. When people hear a trombone solo, they expect to see a trombone in the visual field somewhere. If the image is missing and the scene seems less authentic, viewers can step out of the world very quickly.
Some developers use ambient noises to help drop people into VR environments, and leave dramatic sounds out of the picture altogether. Others use a combined approach, where the soundtrack exists, but it's muted.
VR soundscapes also differ from their traditional counterparts in their lack of scripting. When the viewer steps into the action, anything can happen. "It’s hard to predict what noises an object might make, or where they might be heard," says Tom Abate, associate director of communications at Stanford School of Engineering.
Computer programming can help ease some of the uncertainty, so the right noises happen at the right time. These algorithms are fast, so designing for multiple variables could be easier than ever.
When Does the Tech Work?
In theory, almost any experience could benefit from high-quality 3D sound. But experts say they're not quite sure when it's required and when it's simply nice to have.
"There’s currently no blueprint for piecing together the storytelling that’s best suited for this new type of virtual medium. As far as content goes, there is consensus about only one thing: in a virtual world, it’s all about 'storydoing,' not storytelling," says Eva Wesemann, director of creative strategy for Antenna International.
When you're creating an experience that must be immersive to be appreciated, adding 3D HQ audio makes a lot of sense. And when it's done right, people do have experiences they don’t forget.
Taylor Soper, managing editor of GeekWire, played a game with full 3D audio. "While playing, I didn’t think too much about how the audio was engineered—but perhaps that is the point. Whether it was firing off a bottle rocket that whizzed around my head or jingling a noisemaker near my right ear, the audio technology helped engulf me into this virtual world and made it feel that much more life-like," he says.
Examples of Projects to Inspire You
Understanding where the technology is used right now could help you decide whether or not you need to incorporate those elements into your own projects. These are a few innovative examples of work done recently.
BINCI developed immersive soundscapes for museums in Germany, Spain and Scotland. Each project used head-tracking and localization to transform the sounds as people moved. The projects allowed visitors to do more than just passively consume art. In Germany, for example, people could become part of the scenes depicted on paintings in galleries.
Just as examining a painting is typically passive, so is the act of listening to music. We take in the sounds others make, and while we might dance to it, we don't change the song as it plays.
Artist Tarik Barri wanted to do something different. In his Versum project, listeners become active players within his songs. "I wanted to literally show people what my compositions looked like. I did this by showing them as 3D visual structures through which either the performer or the audience themselves could literally make a musical trip (and therefore become performers themselves)," he writes.
Those immersive musical performances are made for large spaces. But even individuals can benefit from immersive sounds with simple tools like headphones being transformative. You might look at an image on your smartphone or tablet. Then, slip on headphones, and the audio begins to match. The binaural recordings go further than the stereo method because the recording captures and processes sounds like human ears, complete with interaural cues, explains Mona Lalwani in an article for The Verge.
"It’s a simple concept, but when done with high-quality microphones and played over equally high-quality headphones, the effect is eerie: it fools the brain into believing it's hearing the sounds firsthand."
Designing for small screens could mean crafting immersive experiences in video games, apps, and social media sites. Some content creators are designing these soundscapes now on sites like YouTube. ASMR creators record sounds like clapping, tapping and whispering in tracks that move from ear to ear, forward and back, and up and down. Their goal: To relax and soothe troubled, stressed viewers.
When done properly, the experience is physical. "Basically, it feels like the amazing chills you get when someone plays with your hair or traces your back with their fingertips," says ASMR creator Heather Feather.
It's not clear where the future of these 3D soundscapes lies. New equipment and sophisticated programming mean the options for creativity are seemingly limitless. But given the innovation happening now, it's safe to say this is a technology to watch.
Images by: antonioguillem/©123RF.com, viewapart/©123RF.com,stokkete/©123RF.com
