In 2010, Adobe introduced one of Photoshop’s first true ‘smart’ features: A tool called Content-Aware Fill that could intelligently remove and replace objects in a scene.
Today, Adobe is now bringing a version of that tool to its compositing software, After Effects, that does essentially the same thing but on moving video clips, which is far more challenging.
One of the many reasons Adobe’s photography and video software has remained so dominant over the years is the company’s heavy investment in research and development, and the fact that many of the seemingly magical automated tools born from its R&D efforts actually make their way into its software.
Admittedly, when it was initially rolled out in Photoshop CS5, Content-Aware Fill rarely worked as effortlessly or flawlessly as it appeared to in the company’s demos. But over time, and several versions of Photoshop, the tool was refined and improved to the point where, more often than not, its automated corrections became impossible to detect.
Adobe is now bringing Content-Aware Fill to After Effects, its digital compositing software that can be used for everything from creating animated graphics, to integrating CG animations with live footage, to replacing green screens. But intelligently erasing and replacing a portion of a static image is one thing; doing it on a video, across multiple frames of changing content, is an entirely other challenge.
Adobe says it’s made possible by not only the improvements made to the original Photoshop version of the tool over the years, but also by Adobe’s Sensei platform; the company’s AI and machine learning framework introduced a few years ago that now powers many of the company’s apps.
Photoshop CS5’s Content-Aware Fill tool handled all of the image processing on the local machine—which limited its capabilities—but the latest version of After Effects Creative Cloud should be able to pass at least some of the image processing workload to the AI in the cloud, which is ever-improving.
Removing an object from a video won’t be as straightforward a process as it is for a photo. Users will have to create masks that define the region of a video that’s to be erased, either using hand-animated or painted masks, or After Effects’ automated tracking tools. It will be even more challenging for the software, as regions of a frame it uses to fill in holes after an object is removed might not actually still be in frame a few seconds later.
So for optimal results, AE also allows a user to point the software to specific areas of a frame that will be best suited as filler for holes, or users can create reference still frames in Photoshop, that AE will use to guide its own edits.
Content-Aware Fill tool for video will be available for After Effects CC 2019 users starting today after a software update, and Adobe suggests it will be especially useful for jobs like removing rigging or boom mics from shots, erasing unwanted watermarks, or deleting a person from a clip altogether.
Don’t expect flawless results right away, but over time this will undoubtedly become a powerful, and potentially controversial, tool. If you decide to run for office in 2020, there’s probably a few karaoke videos you wouldn’t mind disappearing from.