Beck and Chris Milk's version of Bowie's Sound and Vision with 167 musicians was amazing on its own right, as a linear video. But the 360-degree immersive version with binaural sound and facial tracking truly redefines how we can experience live music remotely.
Whether you like Beck or not, you really have to watch it here. Make sure to use the high definition option with headphones on.
I interviewed director Chris Milk about the project, its challenges and the future of immersive digital experiences:
Gizmodo: How did this project come to be? Chris Milk: I was approached by @radical.media, who was producing the project for HudsonRouge and Lincoln. The initial brief was to try to find a way to reinvent an audience's experience of live music, both at the event and when it's broadcast online. Beck was already involved - though the song wasn't locked down yet - and he was interested in working with a wide array of different musicians.
I've played in bands myself, and I've sat on the floor photographing some of the greatest bands in the world rehearse. The thing that's always struck me is how different the sensory, especially auditory, experience is when you're in the middle of the music with the musicians playing off each other around you, versus when you're in the audience and have a wall of amplified sound coming at you from one direction. So that's where it came from, I wanted to recreate that perspective on a large scale so hundreds of people in a room, and eventually many more online, could be immersed in a phonic environment that they'd never normally have access to.
From there the question was how to execute it in the venue, and the idea evolved for me into a sort of inverted theater-in-the-round approach. Beck is on the centre stage rotating in one direction, the audience is sitting around him on a stage that rotates in the opposite direction, and a huge group of musicians of many disciplines are in a ring around the audience. Sitting in the crowd, the sounds of individual instruments and voices would be coming from every direction around you, changing all the time. I should point out that Willo Perron and Associates were the ones who helped take my initial idea for the staging and figure out a way to physically manifest it all, and more for that matter. Believe me, no small task, and they were incredible in the way they executed it all.
Gizmodo: Why did you choose to go with this multiple panoramic camera technique? Was it a creative decision or a purely technical wow factor one? CM: When you utilise the tools we did, the 360 cameras by 360Heros, the multi axis binaural sound heads, the facial tracking, you get to see and hear the concert with the closest possible perspective to actually being amongst the audience, inside the music. In some respects it's actually better than being there because of the freedom it affords you - you can stand on stage with Beck, or walk around the outside of his stage, or walk through the musicians. All behaviours that at the actual event, would have been severely frowned upon.
Personally, what interests me most is finding new ways to tell stories, using modern and possibly unexplored tools, that will resonate with people on a deeper emotional level. We don't know what the established models of interactive storytelling will be in 100 years, just like the pioneers of cinema didn't envision a 90 minute feature film with a three-act structure. We can only experiment, keep creating new canvases, keep painting new things on them. The best part about this quickly-evolving interactive medium is that the viewer or listener or user isn't a passive receiver anymore: they're participating in the narrative, they're co-creating the art. Look at web-based interactive films, video games, or virtual reality environments - all of them have resonance because they're as much about what the participant says to the piece, as what the piece says to them.
Gizmodo: Were there any particularly strong technical challenges?
CM: From the largest working turntable rotating hundreds of people, to the prototype cameras that have never been used in a major production before, to the 360 binaural recording heads that had to be invented and built from scratch, to the webcam facial tracking system, it would be pretty hard to find a part of this project that wasn't a huge technical challenge. I give large props to my line producer Samantha Storr for holding the many unravelling threads together.
Gizmodo: I can imagine a future in which all live events are filmed like this, probably in 3D too and with perfect camera tracking. How do you see your role, as director, fitting here? Do you think you are giving your power to the audience?
CM: If the narrative, or the experience surrounding it, becomes more compelling for the audience watching it because they feel they have more control, I'm more than happy to give it to them.
Let's be clear though, no matter how much control you give an audience, you are always presenting some sort of curated experience. No story, no matter how simple, or how much control you give to the viewer, is ever going to tell itself. Everything has to be carefully constructed and thought through before hand so that when the user engages with it, the experience seems effortless, seamless, and deeply compelling. That doesn't just magically happen though. The more seamless and in control you feel, the more work that probably went into it before you ever stepped in front of it.
Video games are the most obvious current example of this. These are highly curated experiences that are designed to feel like you are in complete control, and the author of your journey. But there is always a directorial hand guiding it on a multitude of levels, even if you never see or feel that hand.
The last few years I've been doing a lot of research and experimentation in immersive and interactive media environments. A tremendous of time and effort goes into making these sort of projects. If they manage to work, they can be profoundly moving human and emotional experiences. This is the side of technology that we're really only just beginning to unlock, and the part that fascinates me most.
The Wilderness Downtown with Arcade Fire for instance was an experiment to try to create a customised, deeply nostalgic experience for every user. The Treachery of Sanctuary installation was an experiment in absorbing the participant into the piece, creating something visceral and dramatic using just their own simulated shadows on a wall. And so with this Beck project, given the immersive nature of the live performance I was proposing, I wanted the online experience to do the same thing: to mirror the way we look and listen and feel when we're surrounded by such dynamic stimulus. To do that we had to construct an immersive 360 video capture solution and accompanying 360 audio tracks.
Gizmodo: Recently I saw a presentation by a JPL researcher in which he envisioned a near-future of holodecks and 3D goggles to take us in real-time to remote places in the solar system and the galaxy. Where would you like this to go? Both as a creator and a viewer?
CM:I can't wait for that to become a reality. The next thing I'm interested in exploring is pushing cinema in the direction of video games. Where the image feels like a film, and the characters are real human actors, but there is a level of immersion and interactivity that doesn't currently exist in modern cinema. Video games' rendering ability will inevitably advance to the level where they are completely photo-real. And feature films and games will eventually spawn a sort of hybrid class of narrative. But that's a bit down the road, and there are going to be some in between steps first. That's what I'm most exited to try to build right now. How can a film be more than a film. How can we use modern technology to tell stories that resonate on a deeper emotion and human level than possible before that technology existed.
Thank you Chris, and thanks for the outstanding job on Sound and Vision.