You talkin’ to me … in German?
New deepfake technology allows Robert De Niro to deliver his famous line from Taxi Driver in flawless German—with realistic lip movements and facial expressions. The AI software manipulates an actor’s lips and facial expressions to make them convincingly match the speech of someone speaking the same lines in a different language. The artificial-intelligence-based tech could reshape the movie industry, in both alluring and troubling ways.
The technology is related to deepfaking, which uses AI to paste one person’s face onto someone else. It promises to allow directors to effectively reshoot movies in different languages, making foreign versions less jarring for audiences and more faithful to the original. But the power to automatically alter an actor’s face so easily might also prove controversial if not used carefully.
The AI dubbing technology was developed by Flawless, a UK company cofounded by the director Scott Mann, who says he became tired of seeing poor foreign dubbing in his films.
After watching a foreign version of his most recent movie, Heist, which stars De Niro, Mann says he was appalled by how the dubbing ruined carefully crafted scenes. (He declines to specify the language.) Dubbing sometimes involves changing dialog significantly, Mann says, in an effort to make it more closely match an actor’s lip movements. “I remember just being devastated,” he says. “You make a small change in a word or a performance, it can have a large change on a character in the story beat, and in turn on the film.”
Mann began researching academic work related to deepfakes, which led him to a project involving AI dubbing led by Christian Theobalt, a professor at the Max Planck Institute for Informatics in Germany. The work is more sophisticated than a conventional deepfake. It involves capturing the facial expressions and movements of an actor in a scene as well as someone speaking the same lines in another language. This information is then combined to create a 3D model that merges the actor’s face and head with the lip movements of the dubber. Finally, the result is digitally stitched onto the actor in a scene.
Flawless drew inspiration from Theobalt’s project for its product. Mann says the company is in discussions with studios about creating foreign versions of several movies. Other demo clips from Flawless show Jack Nicholson and Tom Cruise delivering famous lines from A Few Good Men in French and Tom Hanks speaking Spanish and Japanese in scenes from Forrest Gump.
“It’s going to be invisible pretty soon,” says Mann. “People will be watching something and they won’t realize it was originally shot in French or whatever. “
Mann says Flawless is also looking at how its technology could help studios avoid costly reshoots by having an actor appear to say new lines. But he says some actors are a little unsettled when they see themselves manipulated using AI. “There’s a fear and wow—they’re the two reactions I keep getting,” he says.
Virginia Gardner, an actress who stars in Mann’s latest movie and has seen herself speaking Spanish thanks to Flawless’ software, doesn’t seem too worried, although she assumes that films modified with AI would include a disclaimer. “I think this is the best way to be able as an actor to preserve your performance” in another language, she says. “If you’re trusting your director and you’re trusting that this process is only going to make a film better, then I really don’t see a downside to it.”
AI video manipulation is controversial—and for good reason. Free deepfake programs that can seamlessly swap one person’s face onto someone else in a video scene have proliferated with advances in AI. The software identifies key points on a person’s face and uses machine learning to capture how that person’s face moves.
