AI Visual Dubbing in Swedish Movie ‘Watch the Skies’

The Swedish UFO thriller Watch the Skies is set to premiere in US theaters in May, but it is generating attention for an unconventional reason: it will be dubbed in English using generative AI technology.

The film, originally titled UFO Sweden, was shot with Swedish actors, but for its American release, the English dialogue will be synchronised to the actors' lip movements via AI.

This "visual dubbing" technique, developed by AI firm Flawless, uses TrueSync machine learning technology to ensure the actors' performances match the new language seamlessly.

Director and writer Victor Danelle views the AI innovation as a way to reach a broader audience, acknowledging the potential concerns from actors and the public about AI's increasing role in filmmaking.

Despite potential backlash, Danelle emphasizes that the use of this technology allows the film to retain its essence while making it accessible to English-speaking viewers.

The project has received the full backing of the SAG-AFTRA, the actors' union, which has expressed concerns over AI's impact on the industry.

Danelle remains confident, stating that the film remains authentic with the same actors and performances, only in a different language.

He said:

“A lot of filmmakers and a lot of actors will be afraid of this technology at first, but we have the creative control, and to act out the film in English was a real exciting experience. It’s still our movie, it’s still the actors’ performance, and that’s the key part.”

Backlash for AI Use in Movies

Scott Mann, co-founder of Flawless, explained in a 2023 presentation that the company’s TrueSync technology uses deep learning to create a 3D model of an actor's face throughout a film.

This model allows the technology to adjust the actors' facial expressions to match the desired dubbed language.

TrueSync was previously used in the 2022 film Fall, where it helped produce a PG-13 version by removing profanity.

Flawless has also developed DeepEditor, a tool that allows editors to transfer an actor's performance from one scene to another without the need for reshoots.

In partnership with distributor XYZ Films, Flawless is also localising several upcoming projects, including The Light, Vincent Must Die, and Smugglers using TrueSync.

While the technology has been well-received in some circles, its use in filmmaking has sparked broader concerns.

A recent example occurred in Netflix's docuseries American Murder: Gabby Petito, where AI-generated voices were used to portray the deceased character.

The decision faced criticism, especially after Petito's parents expressed discomfort with hearing an AI-generated version of their daughter’s voice, despite consenting to the series.

This incident reflects the growing debate over AI’s role in the creative industries and its potential impact on authenticity and respect for the subjects involved.

Say Goodbye to Subtitles in Movies?

AI dubbing technologies, such as Flawless’ TrueSync, offer significant benefits for the film industry by enabling movies to reach global audiences more efficiently, as seen in EWatch the Skies.

These technologies streamline the dubbing process, saving time and resources while enhancing accessibility for viewers across language barriers.

Additionally, innovations like DeepEditor improve the post-production process, creating further efficiencies.

However, there are notable concerns.

AI dubbing may lack the authenticity and emotional depth that human actors bring to their performances, potentially undermining the viewer's connection to the film.

The use of AI in sensitive contexts, such as generating voices for deceased individuals, has raised ethical issues, as seen in the American Murder: Gabby Petito docuseries.

Additionally, the increasing reliance on AI could lead to job displacement for human professionals, further fuelling concerns about the erosion of human artistry.

In a nutshell, while AI dubbing offers considerable advantages in terms of efficiency and accessibility, the film industry must balance innovation with the preservation of human talent and ethical standards to ensure AI’s role remains beneficial without compromising the essence of cinema.