DreamActor-M1 - An AI-powered image animation framework launched by ByteDance

What is DreamActor-M1?

DreamActor-M1 is an advanced AI image animation framework launched by ByteDance, capable of transforming static portrait photos into vivid animated videos. It employs a hybrid guidance mechanism, combining implicit facial representation, 3D head spheres, and 3D body skeletons as control signals to achieve precise control over facial expressions and body movements. It supports multilingual voice-driven facial animation and can generate lip-synced results. DreamActor-M1 boasts high fidelity and identity preservation capabilities, producing videos with strong temporal continuity.

Main functions of DreamActor-M1

• Convert still photos to moving imagesBy combining a still photo and a reference video, the person in the photo is seamlessly replaced into the video scene, generating dynamic images with delicate expressions, natural movements, and high-definition quality.
• Fine controlCombining implicit facial representation, 3D headball, and 3D body skeleton, it achieves robust control over facial expressions and body movements.
• Flexible movement transferSupports transmitting only partial motion, such as facial expressions and head movements.
• Facial animation supportIt can be extended to audio-driven facial animation, enabling lip-syncing for multiple languages.
• Shape-aware animation: Animation generation that adapts to shape is achieved through bone length adjustment technology.
• Diverse style supportIt is robust to various roles and sports styles.
• Multiple perspectives supportedIt can generate animation results under different head poses.

The technical principles of DreamActor-M1

• Hybrid boot mechanismDreamActor-M1 employs a hybrid guidance mechanism, combining implicit facial representation, 3D headball, and 3D body skeleton control signals to achieve precise control over facial expressions and body movements. This hybrid guidance method ensures highly realistic animation in detail while maintaining overall coordination and smoothness.
• Multiscale adaptationThe framework, through a progressive training strategy, can handle images with various body poses and different resolutions, supporting the transformation from portrait to full-body view. This multi-scale adaptability enables DreamActor-M1 to generate high-quality animated content in a variety of scenarios.
• Long-term consistencyDreamActor-M1 ensures temporal consistency in unobserved areas during complex motion by integrating motion patterns and visual references from consecutive frames. This is crucial for generating long, coherent animations and helps avoid common continuity errors.
• Facial animation and voice supportThe framework supports audio-driven facial animation and enables synchronized lip-syncing across multiple languages. Voice input can drive changes in facial expressions, generating natural lip movements and further enhancing the realism of the animation.
• Shape-aware animationDreamActor-M1 possesses shape-aware animation capabilities, enabling shape-adaptive animation generation through bone length adjustment technology. This allows the generated animations to better adapt to the body characteristics of different characters.

DreamActor-M1 project address

• Project official website:https://grisoon.github.io/DreamActor-M1/
• arXiv technical paper:https://arxiv.org/pdf/2504.01724

Application scenarios of DreamActor-M1

• Virtual character creationIt can seamlessly transfer an actor's performance to a virtual character, capturing the actor's facial expressions, movement rhythm, and emotional details, giving the virtual character vivid emotions and movements.
• Personalized animated video generationUsers can transform their photos into lively and interesting animated videos, increasing the fun and appeal of their content. For example, a still photo can be turned into an animated video of dancing, singing, or other fun actions, and then shared on social media.
• Virtual anchors and internet celebritiesIt can be used to create virtual anchors or internet celebrities, reducing production costs and time. Virtual anchors can broadcast live and publish content 24 hours a day, and their appearance and style can be customized according to needs.
• Creative Advertising ProductionAdvertising agencies can use technology to quickly generate attractive advertising videos, combining products with vivid character animations to enhance the appeal and reach of their ads.