Creating Spatial Audio for 360 Video using FB360 Spatial Workstation

Spatial audio is a powerful way to fully immerse a user and direct attention within your VR app via sound. A huge part of our attention can be directed with audio cues but a fully immersive experience requires spatial audio and not just cues. Spatial audio makes what we hear a believable auditory experience that matches what we see and have experienced. It allows you to realistically present audio cues within your application from any direction, drawing the user's attention and providing a believable VR experience.

In this blog we’ll cover the basics of spatial audio and provide a few how-to guides to help you make an awesome VR experience.

The human brain interprets auditory signals in a specific way that allows it to make decisions about its surrounding environment. We use our two ears, in conjunction with the ability to move our heads in space, to make better decisions about the position of an audio signal and the environment the sound is in.

Spatial audio in virtual reality is the manipulation of audio signals so they mimic acoustic behavior in the real world. An accurate sonic representation of a virtual world is a very powerful way to create a compelling and immersive experience. Spatial audio not only serves as a mechanism to complete the immersion but is also very effective as a UI element: audio cues can call attention to various focal points in a narrative or draw the user into watching specific parts of a 360 video, for example.

Spatial audio can be experienced using a normal pair of headphones. No special speakers, hardware, or multi-channel headphones are required. For more details on how spatialized audio works, check out our Introduction to Virtual Reality Audio.

Try it for yourself with these examples of spatialized audio. Make sure you have your headphones on!

While the playback and consumption of spatial audio is the same regardless of whether the experience is an interactive experience, 360 video, or a cinematic mixed reality piece, the workflow to create such content is significantly different. In particular, games and interactive experiences often rely on audio samples played from discrete sound sources that are mixed in real time relative to the position of the camera. The Oculus Audio SDK is designed to add high-quality spatialization tools to existing tools (such as FMOD and Wwise) and engines (Unity and Unreal) often used by game developers.

Designing audio for 360 videos typically requires a very different workflow. Sound designers work with a set of tools organized around a timeline where the final results are predetermined. Though the content is by definition linear and deterministic, this model gives sound designers maximum creative control over the experience. Digital Audio Workstations (DAWs) such as Pro Tools, Nuendo, and Reaper are typically used to craft such content. Currently for 360 video, the most common method of developing a spatial soundscape is a technology called ambisonics.

Ambisonic technology is a method to render 3D sound fields in a spherical format around a particular point in space. It is conceptually similar to 360 video except the entire spherical sound field is audible and responds to changes in head rotation. There are many ways of rendering to an ambisonic field, but all of them rely on decoding to a binaural stereo output to allow the user to perceive the spatial audio effect over a normal pair of headphones.

Ambisonic audio itself can be of n-orders comprising of various channels. More channels results in higher spatial quality, although there is a limit to the perceived difference in sound quality as one goes beyond 3rd order ambisonics (16 channels of audio). Regardless of the number of channels used for encoding the original signal, the decoded binaural audio output will always be to two channels. As the listener moves their head the content of the decoded output stream shifts and changes accordingly, providing a 3D spatial effect.

Ambisonics are not the only way to render spatial audio for 360 videos. There are other solutions in the market as well, although the effectiveness, feature set, toolchains, and final render quality varies between various techniques:

The FB 360 Spatial Workstation is an end-to-end pipeline that allows sound designers to drop in audio sources, pan and sync to scene elements, and render to a single ambisonic file that is played back on Facebook and Oculus video. Originally developed by Two Big Ears, Facebook 360 Spatial Workstation is now a free tool provided by the Audio 360 team at Facebook.

Facebook 360 Spatial Workstation is a collection of plugins for DAWs that include a spatializer, video player, encoder, and rendering SDK, just to name a few. These plugins help authors create spatial audio content, encode it with platform-specific metadata (for Facebook, YouTube, etc.), and play it back in a client application.

Facebook 360 Spatial Workstation is a collection of plugins for DAWs that include a spatializer, video player, encoder, and rendering SDK, just to name a few. These plugins help authors create spatial audio content, encode it with platform-specific metadata (for Facebook, YouTube, etc.), and play it back in a client application.

The diagram above illustrates a typical end-to-end workflow focusing on sound design, asset preparation, mixing with final video, and publishing to Facebook, Oculus or other supported apps.

For most 3rd party apps on Gear VR and other platforms using the Rendering SDK, the sound designer prepares a .tbe file, which is delivered separately. The 3rd party application has the underlying Rendering SDK integrated, which allows it to play back the .tbe file in sync with the video file. There are multiple APIs included with the documentation that allow synchronization to an external clock.

On Facebook and YouTube, the Facebook 360 Encoder application creates an upload-ready .mp4 file.

The Facebook 360 Spatial Workstation is a collection of plugins for creating interactive spatial mixes for 360 videos.

The Facebook 360 Encoder application takes a video file and combines the audio files into the video container, suitable for playback on Facebook and other supported platforms. Additionally, it also allows adding metadata to the file describing values for the Focus feature.

This process also injects relevant metadata into the tool, making the final asset ready for upload to supported platforms.

Here are some key resources to get started with Facebook 360 Spatial Workstation.