In cases of both virtual and augmented reality, what your hands are doing need to be seen and interpreted. If you can’t interact with your hands in a virtual world, you can’t do anything. Say you want to pick up items from a virtual desktop, drive a virtual car or produce virtual pottery. The hands are obviously key.
A new system has been developed which uses a “convolutional neural network” that mimics the human brain and is capable of “deep learning” to understand the hand’s nearly endless complexity of joint angles and contortions.
“We figure out where your hands are and where your fingers are and all the motions of the hands and fingers in real time,” said Karthik Ramani, Purdue University’s Donald W. Feddersen Professor of Mechanical Engineering and director of the C Design Lab.
Ramani created a system called DeepHand, which uses a depth-sensing camera to capture your hand, and specialised algorithms then interpret hand motions.
“It’s called a spatial user interface because you are interfacing with the computer in space instead of on a touch screen or keyboard,” Ramani said.
The researchers “trained” DeepHand with a database of 2.5 million hand poses and configurations. The positions of finger joints are assigned specific “feature vectors” that can be quickly retrieved.
The researchers identify key angles in the hand, look at how these angles change, and these configurations are represented by a set of numbers. Then, from the database the system selects the ones that best fit what the camera sees.
“The idea is similar to the Netflix algorithm, which is able to select recommended movies for specific customers based on a record of previous movies purchased by that customer,” Ramani said.
DeepHand selects “spatial nearest neighbors” that best fit hand positions picked up by the camera. Although training the system requires a large computing power, once the system has been trained it can run on a standard computer.