SCAIL-2 - A character animation model jointly developed by Zhipu and Tsinghua University.

What is SCAIL-2?

SCAIL-2 is the second-generation film-grade character animation framework open-sourced by Zhipu in collaboration with Professor Liu Yongjin's research group at Tsinghua University. Based on a Diffusion Transformer (DiT) architecture, it achieves high-quality animation transfer directly through end-to-end training, eliminating the need for traditional pose estimation intermediates and avoiding ambiguity in skeleton diagrams. It supports both animation generation and character replacement modes, achieving accurate matching through SAM3 mask enhancement. Capable of driving human figures and handling non-human targets such as animals and hand-drawn characters, it supports multi-character interaction, providing efficient animation solutions for film, games, virtual idols, and other fields.

Main functions of SCAIL-2

• End-to-end animation generationBased on the DiT diffusion transformer architecture, it directly bypasses the intermediate steps of traditional pose estimation, and achieves high-quality end-to-end transfer from reference image to driving video action, avoiding skeleton graph ambiguity problems.
• Animation ModeThe appearance of the reference character is completely transferred to the motion sequence of the driving video to achieve character motion replication.
• Replacement Mode: Precisely replace a specified character in a video while preserving the original motion trajectory and background environment.
• SAM3 mask enhancementUsing SAM3 as the reference image and the driving sequence to extract explicit masks as conditional input significantly improves the matching accuracy between characters and actions.
• Diverse driver source supportIt not only supports human motion-driven operation, but can also handle non-human targets such as animals, hand-drawn characters, and robots, and can even control quadrupedal animals without animal training data.
• Multi-character interaction processingIt supports the coordination of character movements and the handling of relative positions in multi-person scenes, enabling complex interactive animations.
• Three-dimensional consistent attitude representationPreserve depth information and occlusion relationships to ensure the spatial rationality of the generated animation.
• Full context pose injectionCombine P-RoPE (Pose-shifted RoPE) to achieve spatiotemporal joint reasoning and improve the coherence of long sequence animations.

Technical Principles of SCAIL-2

• Diffusion Converter (DiT) Infrastructure: Based on Diffusion Transformer, a generative model is built, which utilizes its powerful sequence modeling capabilities to handle long-term animation generation tasks.
• In-Context LearningIt directly utilizes the inherent context learning capabilities of DiT for motion transfer, without relying on pose estimation methods such as NLFPose as intermediate representations, thus avoiding ambiguity issues in skeleton diagrams in complex scenes.
• 3D-Consistent Pose RepresentationPreserve depth information and occlusion relationships within the model to ensure the rationality and consistency of the generated animation in the spatial dimension.
• Full-Context Pose InjectionBy using the P-RoPE (Pose-shifted RoPE) mechanism, pose information is injected into the entire sequence context, enabling spatiotemporal joint reasoning and improving the coherence of long sequence animations.
• SAM3 Explicit Mask ConditionsSAM3 is used as a reference image and driving video sequence to extract precise masks as explicit conditional input, enhancing the accurate alignment between character appearance and motion-driven behavior.
• End-to-end unified trainingAbandoning the traditional two-stage pipeline of "pose estimation → animation generation", it adopts an end-to-end unified training strategy to learn actions directly from the driving video and transfer them to the reference character.
• High-quality data filtering pipelineWe constructed a high-quality training dataset that underwent rigorous screening and manual review, covering diverse targets such as humans, animals, and hand-drawn characters, to ensure the model's generalization ability and generation quality.

How to use SCAIL-2

• Clone the repository and configure the environmentFrom GitHub repository https://github.com/zai-org/SCAIL-2 Pull the code and install the dependencies according to the README (usually including PyTorch, Diffusers, and related vision libraries).
• Download pre-trained modelFrom HuggingFace https://huggingface.co/zai-org/SCAIL-2 Download the SCAIL-2 model weight file and place it in the specified local directory.
• Preparing to input materialsPrepare a reference image and a driving video containing a sequence of target actions. The system will automatically extract explicit masks from the reference image and driving sequence using SAM3 as conditional inputs.
• Select generation modeChoose according to your needs Animation Mode(Transferring the appearance of a reference character to the driving video motion) or Replacement Mode(Replace the specified character in the driving video while retaining the original animation and background).
• Execute animation generation commandRun the script in the corresponding mode in the terminal, for example, using the end-to-end animation mode:python NLFPoseExtract/process_animation_aio.py --subdir <example_dir> --e2e_modeOr use the replacement mode:python NLFPoseExtract/process_replacement.py --subdir <example_dir>.
• Get the generated resultsAfter the script finishes running, the generated character animation video can be retrieved in the output directory and used directly for post-editing or further processing.

SCAIL-2 project address

• Project official website: https://teal024.github.io/SCAIL-2/
• Github repositoryhttps://github.com/zai-org/SCAIL-2
• HuggingFace model libraryhttps://huggingface.co/zai-org/SCAIL-2

SCAIL-2's core advantages

• End-to-end with no intermediate statesAnimation transfer is achieved directly based on the DiT diffusion transformer, eliminating the need for traditional pose estimation (such as NLFPose) as an intermediate representation, avoiding ambiguity in the skeleton diagram, making the process simpler and the results more reliable.
• Film-quality productionIt employs a high-quality data screening pipeline and 3D consistent pose representation, preserving depth information and occlusion relationships, and outputs results that meet film and television standards.
• Dual-mode flexible applicationIt supports Animation Mode (appearance transfer) and Replacement Mode (character replacement), meeting the two core needs of animation generation and video editing.
• Diversified driving capabilitiesIt not only supports human motion-driven operation, but can also handle non-human targets such as animals, hand-drawn characters, and robots, and can control quadrupedal animals without corresponding training data.
• Precise mask alignmentThe integration of SAM3 extracts explicit masks as conditional inputs, significantly improving the matching accuracy between character appearance and driving actions.
• Multi-character interaction supportIt possesses the ability to coordinate character movements and handle relative positional relationships in multi-person scenarios, adapting to complex interactive scenarios.
• Spacetime Coherent ReasoningThe spatiotemporal joint reasoning mechanism is achieved through full context pose injection (P-RoPE) to ensure the temporal coherence and action consistency of long sequence animations.

Comparison of SCAIL-2 with similar competing products

Application scenarios of SCAIL-2

• Film and animation productionIt replaces the traditional motion capture process, quickly transferring character designs to preset movements, reducing animation production costs and time.
• Game developmentIt generates animation resources for game characters, such as walking, fighting, and interacting, or is used for the efficient production of cutscenes (CG).
• Virtual idols and digital humansIt enables virtual anchors and digital employees to engage in live streaming interactions or create short video content, achieving real-time or batch animation generation.
• Advertising and Brand Marketing: Integrate brand IP characters or virtual spokespeople into dynamic advertising videos, and support character replacement mode to quickly adapt to multiple versions of materials.
• Social media content creationTransforming hand-drawn images, emojis, or real photos into dynamic videos for content production and dissemination on short video platforms.
• Education and TrainingCreate animated content such as historical figures reenactment and teaching role demonstrations to enhance the fun and expressiveness of online courses.