In an attempt to get some discussions going in this new group, get some traction, I was woundering if people wanted to talk about all the new AR/XR sound stuff that is being rolled out late this year and at full volume next year?

First off, what is XR, well I think it covers all the "stuff" that VR isn't, primarily for the major volume roll outs in 2018 it's where the cell phone user is interacting in thier environment in a "3D" way.

Here's a rough run down on who is doing what (I btw do a lot of my financial planning based off of cell phone volumes, with everything else aside from cell phones really being niche):

Google Tango - the new tango enabled phones have started rolling out, but the 3d camera's and higher end cpu's are just coming. There are also a huge amount of people working on cloud processing in this area. So far this looks like "traditional" AR. https://get.google.com/tango/

Apple 8 and IOS 11 - so far it looks like all new phones will be AR enabled, and the algorithms in IOS 11 also allow fairly good results with the older cameras. https://www.apple.com/ios/ios-11-preview/

RED Hydrogen - kind of the disrupter, the main player I see that is expanding out of traditonal AR into full XR with things like the holographic displays and enhanced sound features. Main question on RED is if they can do what they say, and if they do can they handle the demand. http://downloads.red.com/hydrogen.pdf

What I'm finding so far is this stuff is really throwing me a curve ball, for example my general "mind set" in immersive audio is that the speakers surround the customer with an attempt to isolate the person from thier setting, all this XR stuff is doing almost the exact opposite.

Some examples to start the discussiion:

1) watching & playing with a dinosaur in your living room. Problably done by kids with something like a ipad, but I can see my grand daughter doing this with a cell phone. On the sound side, the goal is to have the dinosaur sound like it's in the room. A related example could be some squirel game.

2)Having a bird singing on the counter. Good fit for this would be RED, where basically a holographic bird is on the desk singing away. The sound concept would be having the sound mix to the room perfectly, so it sounds/feels like the bird is actually in the room right under the cell phone screen. Related example would be sound design of maybe different bugs or frogs "singing" from the direction of the cell phone.

Just some Idea's to get discussions going. I'm kind of in a quary on what exactly I should be doing, especially I see this big wave but not sure how I can monetize. I think this whole thing needs a lot of sound brainstorming, so I'm hoping the discussion goes a bit random.

I have always been convinced that Augmented Reailty would overshadow Virtual Reailty in terms of adoption, revenue and flexibility. Neither are particularly well suited to the delivery of long form mixed media, and audio is but a tiny component of the whole in the current implementation of Augmented Reality. It's strengths lie outside of entertainment.

Augmented Reality will eventually become one part of a trio, comprising Voice Recognition and Artificial Intelligence, with entertainment not necessarily being the major driver here either.

5G networks, though, will be a major driver, more so than Virtual Reailty, in terms of expanding viewership, and consequently driving revenue gains for media. But, it will also be a major driver in the widespread adoption of technology such as "Internet of Thigs", "smart cities", remote surgery and medical services, fields where Augmented Reality is best suited to be a part, and a problem solver.

The best approach for mixed media is to start with a well written story, employ the technologies that would best convey that story, be that Object-based Media, Immersive Audio, Augmented Reality, and deliver that content over uncapped 5G networks allowing consumption on almost any device the viewer or listener requests, and at any time.

The content is the driver, not the technology. The technology merely assists the content.

For AR, I predict bone-conduction headphones to grow in popularity for reasons of 1) possible integration with personalized binaural spatial audio systems, 2) the ability to mix with sounds in meat space, and 3) social acceptance by other people during interactions that traditional earbuds and headphones don't achieve.

As for audio formats, I'm on the fence between object-based audio and ambisonics (I do development in both currently). From an acoustic standpoint, object-based audio is more directive and can achieve more interesting/compelling/experimental effects than what HOA does. As a point of illustration, you'll need infinite higher-order HOA to achieve an ideal-plane wave without spatial aliasing. Conversely, HOA in its intended use is suited for reproducing and editing captured audio from your typical tetra-mics/audio cameras. This may be good for transplanting someone into different environments but that may not gel well with the requirements of AR (may not blend well with your meat-space acoustic environment).

It's worth clarifying that the usage of the term "object-based" seems to differ greatly according to focus area, and possibly which side of the pond you're on, mostly due to Dolby popularising their usage of the term.

My usage (and those involved with what is poorly termed "next generation" mixed media, and not limited to just broadcasters and OTT providers) involves an implementation where the relationships and associations between components of the film or music mix are defined purely by metadata, from microphone capture to playout - metadata is carried through the entire process, which then specifies this relationship between components, which is constructed at the consumption end.

i.e. a final mix isn't the final mix in the traditional workflow, nor is a picture edit necessarily the final cut, where the ingredients to the "final mix" are only combined at the consumption stage according to user preference, location and environment.

A bit like sending a bunch of lego blocks to everyone (the elements of the "mix"), each receiving an instruction sheet (the metadata), allowing the user (the renderer) to rebuild the image on the box, or to create a unique design.

So a camera angle may be an object, or a particular camera's zoom depth or pan, as are the different languages, or each individual character's dialogue, or the sense of perceived depth of the acoustic space.

This is a very different system to Dolby's Atmos "objects" which are simply panned components placed on top of a traditional fixed speaker configuration final mix. Such a system is very static and does not scale to different listening environments nor to user preferences on the fly, and requires a mix for each target platform.

Object-based media (or metadata specified media), by contrast, has no such restriction since every component of the whole, is specified purely by metadata, and is not fixed to a particular speaker configuration, nor a particular playback system or configuration.

This is why I differentiate between Object-based and Immersive Audio. Object-based audio provides more scope even without Immersive audio, but combined the two concepts provide much more flexible and customisable experiences for the consumer of the content.

Indeed, the term "object-based" caries a loose definition due to the absence of standardization in the rendering process. As for meta-data, the MPEG-H format has adopted this idea for transmitting both ambisonics field descriptors and positional data of source-objects. However, I believe the the descriptors are incomplete due to incongruities between the formats. e.g. ambisonics assumption of a fixed sweet spot precludes the movement of the listener in the world-space (field is relative to the listener, can't be easily expanded for translation movements) whereas object-positional information can be independent of the listener position. Other aspects such as early/diffuse-reflections and reverb are ignored entirely (left upto the decoder?).

More generally, I'm somewhat concerned over how a meta-data format would erode artistic controls as it concedes more of the final output to the quality and/or specifications of the renderer further up the pipeline.

It may in fact provide more long term creative options, as the idea of a "living work" is somewhat possible, where an alternate or enhanced versions may only require an update of the metadata and not necessarily all of the components.

The renderer is dumb, it merely does what the metadata specifies. Once the ITU finalises the "base" specification (akin to AES67 providing the "base" specification for interoperability between Audio Over IP standards), the media landscape will be less closed off from the different providers and implementations, eliminating silos.

One reason object-based audio is such a massive change from what has come before, is that whether the content is stereo or a full immersive theatre presentation, the system remains the same. The benefits of such a dynamic system are present even when listening in mono on a standard car radio or employing all 64 speakers in a high-end Audi.

To return to the original topic, the results from this Avid survey are somewhat interesting:
https://www.ibc.org/content-manageme...-/2020.article

This isn't so as there's large variance in the final output due to ambiguity in how to decode metadata of say a high order ambisonics field or even playback a moving point-source (non-VBAP systems). The former is an overconstrained systems (no unique solution to resolve sound-pressure and velocity constraints at the sweet spot), no specifications on handling field-transformations due to say head-rotation. The latter runs into various phase issues due to moving head/source/listener.

While the lamentations of musicians against teens listening to their works through 10$ earbuds are duly noted, a meta-data format will move the problem further up-stream into software land. On the flip-side, the ambiguity here may open up opportunities. An audio-script can be performed in many ways.

The object-based audio renderer is so much larger than ambisonics which isn't necessarily a part of the production or delivery system.

Your totally right on the story, it's been real tough so what I started to do out of frustration more then anything else is story board the Sound Design, I included my sound scape "manga" type drawing which I'm doing through the entire piece.

On the adobe survey, that's somewhat similiar to the VR surveys, where the main thing I'm having a problem with is figuring out how to monetize. It's been a struggle, and I've moved almost all my finishing over to unity as a somewhat shot in the dark. I do have a lot of background in monetizing multimedia through transitions, so when I have problems I can relate to people that are just kind of giving up on this stuff. On the other hand the numbers, like what you said way above, are just so compelling that giving up on XR would be like trying to avoid the digitization of music.

I've really noticed what your talking about with the "meat-space", what your pointing out is the sound physics of xr is totally different then normal immersive sound. For example, if I have a voice actress in my physical room, the immersion comes from forgetting she is actually not in the room with speaker arrays. With normal immersive vr and things like 5.1, i'm trying to get the listener isolated from the room. From a pure energy/cost viewpoint, XR is much cheaper to do then vr or normal 5.1 since the cost of sound isolation is very expensive physically.

Yep, XR would require good estimations of the meat-space's acoustics so that either a virtual soundscape can blend ontop of it or the meat-space actors can be augmented to fit the virtual soundscape. There're some ongoing research in estimating room acoustics from geometry/point-cloud data which may someday intersect with the big-data community but I don't know if this will achieve the fidelity required for immersion. That said, have you observed a larger propensity of an acoustic uncanny valley in XR?

In the couple or hundred audience tests I've done I have found a huge divergence in the uncanny valley:

1) Passive XR - Almost in every case people say they are looking at a "ghost" world. Especialy problems if one of the ghosts in the xr space moves through the person in real space. I have been studying a lot of anime and also plays, pondering this issue. At this point in time, I think visuals need to be either stylistic or in a stage "play" arena. In either case the story has to match to a XR medium, there is no way I can see just putting a standard story into XR?

2) Active XR - I've been surprised here how the uncanny valley is somewhat "forgotten" by the participant if they are interested in the active XR (when I mean active they are doing something like moving blocks or following something). I'm not sure what the psychology is, but my gutt feel is when someone is "hunting" for something they are focused on the sound as information to decision, and just get kind of immersed in the circumstance and ignore the inconsistency.

I haven't done hardly any of this for nature based experiences, but in the passive XR if it's nature based like a bird singing or bug chirping, the imagination of the person seems to be filling in on the gaps in the sound technology. But if it's pure passive looking around watching something in thier own space, the story better be perfectly aligned with the experience or the sound issues are jarring. I also find in passive xr for background noise the person can be hooked to it for hours, but if they are holding something like the pad or cell phone the person normally puts it down under a minute.

I think 2) heightens the role of sound as a means to an end (or a set of cues) in what would be a more interactive medium than a flat screen. Head-tracking in particular tends to heighten the spatial qualities of sound over timbre which allows it to take over the orienting role for guiding the participant, whether be for purposes of exploration or the completion of a task/goal.

For more passive experiences (like 360 video), it feels like being taken for a ride ("hapless victim" role) as the ability to freely rotate the head but not move the body imitates the restricted movements of a paraplegic or an infant. The lack of exploration & task/goals in that setup is probably why the focus of attention is drawn to the aesthetics itself which is where the uncanny valley emerges. If this is the case, maybe 360 video is paradoxically more suited for stylized experiences rather than "meat-space" recorded content.

Victoria Alonso said something similiar to what your saying when I went to a VR conference:

VR on the lot - Victoria Alonso

I really like how she thinks about the problem and her approach to solving the problems through story. This interview btw, for those lurking on this channel, is in my opinion totally mandatory for anyone in the post industry to watch.