One of the obvious tells that a computer-generated character isn’t real is the lack of interaction with the background they have been added to. It’s part of the reason why augmented reality apps are fun, though far from convincing. But researchers at MIT’s CSAIL Lab have come up with clever a way to make background footage interact with a virtual character.
The key is to first capture how physical objects in a scene actually move, but all that’s needed is a few seconds of video recording their near-invisible vibrations for an algorithm to then simulate more vigorous movements.
For example, to allow video of a small wire figure to be manipulated and warped in real-time afterwards, during the recording phase a nearby thump on a table creates enough vibrations in the figure for the algorithms behind this research to determine how it would normally move.
So what does this mean for those us outside academic circles who aren’t thumbing through research journals on the weekend? There are some legitimately cool applications for this technology, particularly now that augmented reality is a popular buzzword again after the introduction of Pokemon GO.
Imagine future versions of the game where a Caterpie appears on a small bush, and actually causes the leaves to bounce and move as it hops around. It would add a new level of realism to the game, and to the countless other AR games that are surely in development now after Pokemon GO’s immense success. The research could also lead to new developments in other industries where computer-generated imagery and live action footage are mixed. It could make creating sophisticated visual effects easier and cheaper for amateur filmmakers working on a tight budget. And who knows, one day it might be the key feature that makes consumers want to actually buy AR devices like Google Glass.
[MIT News]