When I watched Star Wars: A New Hope for the first time as a five-year-old, my face looked like O_O for two straight hours. Spaceships! Lightsabers! Robots! But one piece of future that wormed its way even deeper into my brain was the Millennium Falcon’s holographic monster mash chess game, aka Dejarik. It was the coolest.
This was how I wanted to play games always and forever — not trapped within the limited frame of my television — and Microsoft agrees with me. With its new Hololens 3D experiment, Microsoft details its motion capture process, using 106 RGB and infrared cameras on a green screen backdrop to create what Microsoft calls “high quality, free viewpoint video.”
So yeah, basically Dejarik. Here’s the technical mumbo jumbo of how this all comes together:
Our system records performances using a dense set of RGB and IR video cameras, generates dynamic textured surfaces, and compresses these to a streamable 3D video format. Four technical advances contribute to high fidelity and robustness: multimodal multi-view stereo fusing RGB, IR, and silhouette information; adaptive meshing guided by automatic detection of perceptually salient areas; mesh tracking to create temporally coherent subsequences; and encoding of tracked textured meshes as an MPEG video stream. Quantitative experiments demonstrate geometric accuracy, texture fidelity, and encoding efficiency. We release several datasets with calibrated inputs and processed results to foster future research.
In many ways, that just sounds like good ole mo-capping, except instead of gathering data and then building a 3D model, Microsoft’s technique is way more efficient and just captures and makes the model at the same time.
Looking at those mind-blowing Minecraft Hololens demos, a new generation of UI and gaming seems to be quickly approaching, so let me give you your first piece of advice: Let the Wookie win.