MARZ Unveils LipDub AI to Eliminate Mismatched Lip-Syncing of Dubbed Content for Global Distribution

Using AI technology to match facial movements to dubbed languages redefines the localization industry for content producers and advertisers

Emmy-nominated visual effects (VFX) studio and AI technology startup Monsters Aliens Robots Zombies (MARZ) unveiled LipDub AI, its second AI for VFX product. LipDub AI is a state-of-the-art lip-syncing solution that dramatically improves the quality and appeal of dubbed content, and will transform the entertainment industry’s localization sector.

Read More about AiThority Interview: AiThority Interview with Rebecca Jones, General Manager at Mosaicx

“With LipDub, we have succeeded in leveraging advanced generative AI to create photorealistic, lip-synced videos that are dynamic and temporally consistent”

LipDub AI is the world’s first completely automated lip-sync technology to achieve Hollywood-caliber results. In addition to delivering market-leading quality of visuals, LipDub is a scalable solution that visually transforms one minute of video footage into any language in seconds and in under 20 minutes of compute time for Hollywood-grade outputs.

The proprietary LipDub AI technology uses deep neural networks to seamlessly sync actors’ on-screen lip movements to any dubbed audio file, providing an elevated viewing experience to audiences worldwide watching non-native content. This groundbreaking approach to dubbing content is expected to help Hollywood studios and advertisers drive meaningful leaps in engagement and viewership. Moreover, it will enable creatives to create globally appealing content that truly transcends borders.

According to the MARZ executive leadership team, Jonathan Bronfman – Chief Executive Officer, Matt Panousis – Chief Operating Officer, and Lon Molnar – Chief Creative Officer, “The current standards for passable localization are far too low. Viewers around the world deserve to watch foreign-made content that looks great, just as studios and advertisers endeavor to create content that has global appeal. LipDub AI represents the next logical step in the evolution of dubbing, and it’s profoundly rewarding for MARZ to be spearheading that change.”

AiThority Interview Insights: AiThority Interview with Bret Greenstein, Partner, Data & AI at PwC

The LipDub research team is led by MARZ Chief Scientist and world-renowned professor at the Department of Computer Science at Tel-Aviv University, Danny Cohen-Or, an ACM fellow and The Isaias Nizri Chair in Visual Computing. He leads the MARZ AI research team alongside longtime collaborator Ali Mahdavi-Amiri, Assistant Professor at Simon Fraser University. Together they successfully conducted the research for Vanity AI, the first AI solution developed by MARZ. Vanity AI was launched in January 2023 and has already been used in over 45 Hollywood productions.

“With LipDub, we have succeeded in leveraging advanced generative AI to create photorealistic, lip-synced videos that are dynamic and temporally consistent,” said Cohen-Or.

 Latest AiThority Interview Insights : AiThority Interview with Dan O’Connell, Chief AI & Strategy Officer at Dialpad

 [To share your insights with us, please write to sghosh@martechseries.com]