Researchers are teaching machines to get stuff done using video simulations, a database of chores, and a virtual home reminiscent of your favourite time-wasting video game. The end goal? Teaching robots the same way you teach yourself how to install a toilet: Instructional videos.
Screenshot: MIT CSAIL
Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), the University of Toronto, McGill University, and the University of Ljubljana released a paper detailing the methods by which they taught computers how to accomplish a greater range of activities by watching instructional videos.
The researchers used simulated videos with virtual human characters, along with a database of 3000 crowdsourced tasks the program can choose. The AI then mimics the tasks – along with everything that task entails – seen in the video.
Image: MIT CSAIL
The researchers created video simulations set in a furnished home (with a living room, kitchen, dining room, bedroom and home office), surprisingly similar to houses in The Sims. The artificial agents would watch the videos and attempt to execute the tasks demonstrated. Researchers have so far successfully executed about 1000 of the available crowdsourced actions.
As for learning new tricks, it’s certainly possible, “as long as the task is described as a program with a series of steps that it can understand,” according to MIT CSAIL’s Adam Conner-Simons.
Turning on the TV is easy for a human to understand, but the simple command lacks the instructions a robot would deem necessary in order to execute the task. You can’t turn on the TV if you don’t hit the power button; you can’t hit the power button unless you’re in front of it; you can’t be in front of it until you walk over to it. You get the idea.
Eventually, researchers hope to teach robots how to accomplish tasks simply by showing them actual instructional videos you might find on YouTube, for example. It also means you could eventually talk to your in-home smart speaker, instructing your Google Assistant on how exactly to dim your lights, play your tunes, and set the mood for dinner without manually entering each step.
When I asked Conner-Simons about real-world applications, I suggested a robot could help someone crack open a cold one with the boys. He said that “isn’t exactly the first use case that the team had in mind”, but the ability to move household items would be a valuable skill.
“We envision that a system like this could have important implications for people with limited mobility, such as the elderly or the disabled,” he said.
But what about the lazy?