On the 5th of January 2022, sound designers and sound artists, Felix Deufel and Nikhil Nagaraj facilitated a day-long workshop at the Indian Sonic Research Organization (theISRO), Bengaluru. After touring India and Germany with their latest 3D audio production/performance If we Vanish, Felix and Nikhil wanted to pass a few of their skills on to empower students, sound artists, producers and other practitioners to work with spatial audio.
I have been working at theISRO for a few months, but have little experience with spatial audio. I had the privilege to attend Felix and Nikhil's workshop and learn from them personally. This blog post details some of the things I learned during the workshop.
This blog post was written by Dhruva Gowda Storz
Foundations of 3D audio
The workshop started with an introduction to 3D audio, where Felix and Nikhil explained the basic principles of 3D sound in the context of natural human hearing systems. They described how our biological binaural system (meaning two ears) is capable of perceiving sound in a spatial manner. They explained how various parameters such as the distance between ears, the shape of the head, and the morphology of the structures around our ears (the pinnae) ultimately influence how we perceive 3D audio and the unique frequency spectrum that each person is attuned to.
They then went on to talk about different 3D audio formats such as cinema surround sound, Atmos, Ambisonics, etc. and how these try to simulate natural 3 dimensional sound through different sets of algorithms / mathematical formulae that account for the aforementioned parameters.
Overview of Software
After this introduction, Nikhil and Felix introduced us to the main softwares and plugins used for spatial audio production.
At the core of most advanced sound projects such as this is a digital audio workstation (DAW). Popular DAWs include Logic, Ableton Live, and Reaper. Nikhil and Felix used Reaper as a base for the workshop. DAWs let you setup multiple sound files in a workspace, and manipulate and arrange them as you wish. More importantly, these workstations let you route each sound file to various output channels. At it's core, this is the primary challenge of spatial audio.
It is technically possible to manually select which output channels to route each of your sound files to specific channels based on your arrangement of speakers in your spatial array, but this is a painstaking task and does not account for the specific positions of your speakers in space and how a listener within the spatial audio array will experience sound. This is where Virtual Studio Technology (VST) plugins come in. VST plugins are tools that are compatible with most conventional DAWs and can augment their functionality. In this workshop, we were presented with two open-source plugins developed by the Institute for Electronic Music and Acoustics (IEM) for ambisonics, binaural and multichannel audio that handle the heavy-lifting behind routing for spatial audio, as well as a plugin made by Felix for his practice.
Spatial audio setups typically operate by using an encoder and a decoder. Encoders convert your collection of sound files in your DAW into a common format used for spatial audio, where each sound file is assigned a position in space. During the workshop, we were introduced to IEM's MultiEncoder plugin.
The MultiEncoder plugin can be used to assign positions to each of your sound channels in space (we use 16 as shown in the image above). Each sound channel is numbered and can be positioned on the surface of a hemisphere using the circular interface within the plugin. The circle represents a hemisphere as seen from above, with the center of the circle representing a position directly above the center of the hemisphere. The advantage of using an encoder such as this is that rather than playing audio from a single speaker based on manual output routing, you have more granular control over the direction of your sound. The encoder specifies positions in space rather than specific features, and therefore deals directly with what the listener experiences rather than what the hardware needs to do.
While the encoder gives you granular control over positions in space, your spatial audio blueprint needs to be realized in the real world. The blueprint needs to be appropriately routed to specific speakers, and the volumes of these speakers need to be manipulated to simulate sound coming from the specified direction. This is handled by the decoder. During the workshop, we were introduced to IEM's AIIRADecoder.
AIIRA's interface includes a polygon where each vertex represents a speaker in your setup. The positions of these vertexes can be edited to reflect the positions of your speakers in the real world, ensuring that the encoded spatial audio blueprint is accurately simulated in the real world. The benefit of using a split encoder-decoder setup is that you have more freedom in your practice. For example, you can design a spatial audio piece at home on your personal setup or headphones using one decoder, and can then use the encoder's output file during the live performance with a decoder that corresponds to a different spatial audio array at the venue.
One big limitation of existing spatial audio encoder plugins is that you cannot move your spatial audio channels through space. While the existing plugins work well for static soundscapes, spatial sound artists like Felix find that the fact you cannot dynamically move the channels in the encoder interface around a limitation. To solve this problem, Felix and his organization Not a Number GmBH are developing a VST plugin called Grapes. We got an early preview of this software. Grapes lets you preprogram motion in your soundscape, which lets you, for example, simulate a car moving from one side of the room to the other. Grapes ties into MultiEncoder through the OSC receiver so it's outputs can be visualized in the encoder layout, as can be seen in the image below. Grapes is currently still in production so we couldn't show you how it works, but it will be available to the public soon.
Felix and Nikhil explained the various intricacies of these plugins to us, and how they are each useful when working with spatial audio. Further, we also got an overview of the hardware such as the audio interfaces, amplifiers and speakers, their signal flows and how they work in conjunction with the 3D audio software. We then had a showcase of some of Nikhil's and Felix's work in spatial audio on our lab's array, such as a natural soundscape of a thunderstorm recorded in the rainforests of Coorg, India. It was like we were being transported to another place.
The attendees of the workshop received a folder with all the necessary plugins, software, and instructions, along with an early access copy of Grapes to use for themselves. Felix and Nikhil also helped some of the musicians set up spatial audio pipelines on their own preferred DAWs and systems. As someone new to spatial audio, I found the workshop incredibly interesting and well presented. Here are a few comments by other attendees:
"Felix was really patient with all of us, and although the content taught was a bit technical and distant, he made sure that we could all grasp the step by step process of what he and Nikhil were teaching. They even provided us with the software and plugins! Overall, it was a wonderful experience."
"I was mesmerised with how audio & sound effects as experience(s) can be used to express as well as demonstrate innate artistic creativity. During the workshop, I was astonished with Felix's in depth technical expertise in audio particularly with multi channel audio systems along with Nikhil's artistic vision & process involved in digital audio workstation."