Audio Spatialization + Remote Collaboration

I’ve been thinking for a while about audio spatialization in the context of remote collaboration¹.

Audio spatialization is when you process audio in such a way that it sounds like it’s coming from a location in space, around you, rather than from your device loudspeaker or kind of ‘inside your head’ when you’re using head-/ear-phones. The effect is easier to achieve with -phones because then you have independent control over what each ear hears and no interference between the two; but you can also do it with an array of well-placed speakers, like a home theater surround sound system, and it sounds more natural because natural audio usually doesn’t materialize directly inside your ear (‘bzzzz’ – ‘aagh get out!’ – <a vague disgust lingers>).

The processing is a little complicated², but it is certainly possible to get quite realistic, and modern technology makes it not that computationally challenging either. (Although virtual reality research has explored this topic since at least 1993. Really there was so much VR research in the 90s—sadly, then nor now, the rest of the technological world hasn’t made it commonplace.)

Tying this into remote collaboration: it often happens over group video or voice calls, and the majority of widely-used platforms don’t do spatialization at all³. What would spatialization look like in this setting? My proposal is that each person in a call would have a seat at a virtual table, and would hear the others talking as if from their positions around the table. (This can be optionally augmented by tracking the direction of the listener’s head so you can ‘look at’ someone who’s speaking and the audio changes accordingly. Could use the webcam stream or mobile phone accelerometer data or something else.) Depending on the nature of the work/collaboration, there could be multiple tables within earshot of each other, or each person could have their own table where others can join if they want to talk (and this will be faintly but not distractingly audible to others).

For now let’s stick with the single table. Why might a collaboration platform want to support this? I conjecture:

1. The ‘cocktail party’ problem: real-life discussions can organically generate side-discussions and cross-talk. These are hard to manage in a superimposed mono stream of everyone’s audio, but our brains have a lot of experience in focusing on some (spatial) sources of sound and tuning out others.
2. Keeping track of the conversation without video: if you can’t or aren’t using video, using positioning information can make it easier to tell who’s talking, in addition to recognizing their voice. It’s easier for someone to jump in when their half-word–interruption isn’t mistaken for noise; it’s easier for others to tell that they haven’t spoken for a while.
3. Aesthetics: it just seems more natural. Like actually being at a table together. If not practical value, it has a sort of artistic value⁴.

Of course there are good reasons that major platforms have for not implementing this, probably a big one being computation overhead, either on their servers or on users’ machines, the latter which may not always be capable and the former which may not be worth the (perceived) benefit. But then I discovered that someone has already started to work on just this: High Fidelity. They’re in a beta phase right now so I signed up for a trial, and tried it out with some friends, but it didn’t quite work. (Edit: I’ve since learned from my friends of a bunch of platforms that do this, including Hubs and CozyRoom.) Interested in seeing where this technology goes in the next year or so!