Skydio was one of the first companies to create a flying drone that can autonomously track, follow and frame the action, but it was more concerned about not crashing than aesthetics. Researchers at Carnegie Mellon University are taking the idea one step further by teaching camera drones how to not only fly safely but also ensure they’re capturing the most compelling footage possible.
Job number one for an intelligent drone is simply staying airborne and avoiding obstacles like trees or buildings that could knock out its spinning propellers. A collision doesn’t necessarily mean a crash, many drones can safely land with one broken propeller, but it will put a quadcopter out of commission for a while if repairs can’t be immediately performed.
As a result, while autonomous drones can keep a target in frame, they make compromises for safety, resulting in footage that might not be as dramatic, or aesthetically pleasing, as it would be a human pilot at the controls or with a cinematographer guiding the shots.
The software that controls the researchers’ drone, featuring upgrades that include a LiDAR scanner for mapping the terrain in 3D, was trained using a technique called “deep reinforcement learning.”
A study was conducted where people watched footage generated by a photo-realistic simulator that continuously altered the framing and position of the virtual camera shooting the scene. The test subjects were asked to evaluate and score the various scenes based on his visually appealing or interesting they seemed.
The results of the study were used to create guidelines for the autonomous camera system to follow, so based on the positions of the actors (or other subject matter) as well as obstacles, it’s able to intelligently position the camera, adjust the framing of the shot, and even recreate sweeping camera movements, to generate compelling footage for the viewer.
The most obvious application of the research is to make filmmaking tools more accessible and more affordable so that any one with a story to tell can potentially create a film without needing access to hundreds of millions of dollars. But the intelligent system, which is particularly adept at avoiding obstacles while keeping the most important subject in frame, could help automate other uses for drones including remote inspections or search and rescue missions which typically require a pilot to remain at the controls the entire time.