Researchers at Facebook AI Research and Tel Aviv University school of computer science published a preprint paper outlining how they created an automatic process to create VR avatars. The company’s new Facebook Spaces “social” VR venture launched this week.
Image: Facebook
The process of how Facebook set out to solve the problem of automatically generating VR avatars based on photos is fascinating, but what’s even more interesting is the method used to help train its AI to create VR avatars: It used and mimicked avatars created using Snap Inc.’s Bitmoji service as part of its dataset. The researchers first transformed a set of celebrity photos into 2D avatars and compared the results to manually created Bitmoji avatars of the celebs, then the Facebook-created 2D avatars were transformed into VR avatars.
The paper explains the way Bitmoji was used in the process:
The proposed TOS [Tied Output Synthesis] method is evaluated for the task of generating specification-compliant emoji. In this task, we transfer an “in-the-wild” facial photograph to a set of parameters that defines an emoji. As the unlabeled training data of face images (domain X ), we use a set s of one million random images without identity information. The set t consists of assorted facial avatars (emoji) created by an online service (bitmoji.com). The emoji images were processed by an automatic process that detects, based on a set of heuristics, the center of the irises and the tip of the nose [22]. Based on these coordinates, the emoji were centered and scaled into 152 x 152 RGB images.
The emoji engine of the online service is mostly additive. In order to train the TOS, we mimic it and have created a neural network e that maps properties such as gender, length of hair, shape of eyes, etc. into an output image.
The researchers say that the “emoji engine of the online service is mostly additive”, and the team mimicked the way Bitmoji works to help automatically create avatars.
You can see the Bitmoji the researchers created, side-by-side the avatars the Tied Output Synthesis (TOS).
Image: arxiv.org
Based on the language used, it doesn’t appear as if the Bitmoji avatars were used in the machine learning process — that is, the system wasn’t trained based on the Bitmoji the team manually created for each photograph — instead, the team used the Bitmoji as a comparison data point when trying to make the avatars look as accurate as possible. (As an aside, it’s very amusing to see what celebrities Facebook used for its image samples. I haven’t thought about Shane West in years.)
Still, given how much copying Facebook is already doing of other Snap features (Miranda Kerr is not pleased!), it’s hilarious to see that company’s products in an academic paper touting Facebook’s advanced methods for automatic avatar creation.
[h/t Dave Gershgorn]