Oculus’ New Experimental API Blends Virtual Reality With Your Real-World Surroundings

Oculus’ New Experimental API Blends Virtual Reality With Your Real-World Surroundings
Photo: Josh Edelson, Getty Images

An update for Facebook’s Oculus Quest 2 virtual reality system will let developers incorporate real-life video from the VR headset’s sensors into their games to create “mixed reality experiences.”

With Passthrough API Experimental, Oculus’ new application programming interface, developers can customise how a player’s surroundings appear through their VR headset, applying effects and filters and even rendering the real world onto certain in-game surfaces. The API will roll out first to Unity developers in an upcoming software update, “with support for other development platforms coming in the future,” Oculus VR said in a blog post Friday.

It’s easy to imagine a slew of cool ways games could incorporate your physical environment into gameplay. One gif that Oculus shared of the API in action shows a player drawing on the walls, immediately calling to mind the paint-based turf wars a la Splatoon.

Gif: Oculus VR (Facebook) Gif: Oculus VR (Facebook)

The ability to toggle the mixed reality’s opacity, aka how much of the real-world or virtual one you see at any given time, that’s shown off in another gif could handily be integrated with some kind of puzzle-solving mechanic.

In-game enemies could also hide behind your furniture for sneak attacks. I’m a total chicken when it comes to horror games already, so the idea of monsters being able to jump out from behind my own living room couch makes me want to cry.

Gif: Oculus VR (Facebook) Gif: Oculus VR (Facebook)

The announcement included several examples of uses for the API beyond gaming as well. Oculus posited that it could improve productivity and enable more collaborative teleworking by incorporating workers’ real-life keyboards and desks. Users would also be able to engage with virtual content without losing the ability to interact with their housemates or pets.

When asked if first-generation Quest users can expect to have access, Facebook told the folks at UploadVR that the API is only coming to the Quest 2. A pared-down version of the Passthrough technology is already available on Facebook’s Quest, Quest 2, and Rift S headsets, which lets users take a peek at what’s happening around them while still wearing the headset.

Gif: Oculus VR (Facebook) Gif: Oculus VR (Facebook)

In its announcement, Oculus added that it designed the API “with privacy in mind.”

“Apps that use Passthrough API cannot access, view, or store images or videos of your physical environment from the Oculus Quest 2 sensors. This means raw images from device sensors are processed on-device,” the company said.

As for when developers can expect to ship their games that use Passthrough to players, Oculus said it’s aiming to roll out a production version “later this year.”