Wanxiang 2.1 - The latest video generation model from Tongyi Wanxiang

What is Wanxiang 2.1?

Wanxiang 2.1 is an upgraded version of Tongyi Wanxiang launched by Alibaba. Based on its self-developed high-efficiency VAE and DiT architecture, it enhances spatiotemporal context modeling capabilities, supports efficient encoding and decoding of unlimited-length 1080P videos, and for the first time achieves Chinese text-to-video generation functionality. The upgraded Tongyi Wanxiang has topped the VBench leaderboard. Wanxiang 2.1 can stably display complex human movements, realistically reproduce real-world physical laws, generate Chinese and English video effects with one click, and possesses powerful capabilities for film-quality visuals and artistic style conversion. Wanxiang 2.1 supports text-to-image grouping, adopts the IC-LoRA image generation training method, enhances the contextual capabilities from text to image, and easily achieves the combined generation of related images.

Main functions of Wanxiang 2.1

• Video generation
  • Complex action displayIt can stably display various complex human body movements, such as spinning, jumping, turning, rolling, etc., as well as camera movement, making the video content more vivid and realistic.
  • Physical Law ReductionIt realistically recreates the physical laws of the real world, such as collisions, bounces, cutting, and compression. For example, it can generate scenes of raindrops splashing on an umbrella, making the video more realistic.
  • Chinese and English video effects generationIt offers a variety of video effects options, such as transitions, particle effects, and simulations, and can generate Chinese and English video effects with one click, enhancing the visual expressiveness of videos.
  • Artistic Style TransformationIt possesses powerful artistic style expression capabilities, and can convert the cinematic texture and artistic style of videos with one click, such as film color tone, impressionistic brushstrokes, abstract expression, etc., to generate videos of various styles.
• Graph generation
  • Storyboard effect restorationThe film recreates cinematic-quality scene composition, ensuring seamless integration of elements such as characters, appearances, actions, environments, and lighting, perfectly combining storyline, visual effects, and atmosphere.
  • Four-panel comic strip creationBased on the user's description of the comic's plot and style, generate a four-panel comic to quickly tell a short story.
  • Creative Avatar CustomizationWe offer customized couple avatars, individual avatars, and more, based on users' preferences, styles, and characteristics. We provide a variety of styles to choose from, including cartoon, realistic, and creative styles, to meet diverse user needs.

Technical Principles of Wanxiang 2.1

• VAE architectureVariational autoencoders (VAEs) are generative models that use an encoder to map input data to a latent space and a decoder to map the representation of the latent space back to the data space, thereby generating and reconstructing data.
• DiT architectureDiT (Diffusion in Time) architecture is a generative model based on the diffusion model. It gradually introduces noise along the time dimension and gradually removes the noise to generate data. DiT can effectively capture the spatiotemporal structure of video, supporting efficient encoding and decoding and generating high-quality video.
• IC-LoRAIC-LoRA is an image generation training method that enhances the contextual ability of text to images by combining image content and text description, making the generated images more consistent with the user's text description and expectations.
• Context modelingBased on enhanced spatiotemporal context modeling capabilities, it can better understand and generate video content with coherence and consistency, making elements such as actions, scenes, and styles in the video more natural and coordinated.

Project address of Wanxiang 2.1

• Project official websiteAccess the Tongyi Wanxiang AI Video website on your computer to start creating videos.
• Open source addressWan2.1

Wanxiang 2.1 Video Case Study

Text effects

• Prompt 1: Against a backdrop of red New Year's rice paper, a drop of ink appears, slowly spreading and diffusing. The edges of the characters are blurred and natural. As the ink spreads, the character "福" (fortune/blessing) appears on the paper, with the ink transitioning from dark to light, presenting a unique oriental charm. The background is sophisticated and simple, reminiscent of magazine photography.

• Tip 2: Wide shot, on the banks of the Seine, with dazzling fireworks blooming in the sky. When the fireworks form the pink number "2025", the camera zooms in for a close-up, and then gradually fades away.

sports

• Clue 1: A car speeds along a snow-covered highway. The camera shoots from above, showcasing the trees lining the road and the distant mountains. The car travels along the straight road, creating a striking contrast between the surrounding snow and trees. The entire scene appears exceptionally clear in natural light, possessing a documentary-style realism.

• Clue 2: A massive ocean wave, a surfer leaps from its crest, completing an aerial somersault. The camera emerges from within the wave, capturing the moment sunlight filters through the water. Water splashes in perfect arcs, the surfboard leaving trails as it cuts across the surface. The final shot freezes on the surfer's perfect moment as they pass through the curtain of water.

Special effects

• Clue 1: A giant dragon hovers above a burning city, the afterglow of the flames reflecting on its scales. The camera looks up from below, capturing the dragon's silhouette appearing and disappearing in the flames and smoke. Its wings flap in the air, stirring up a gale that blows the flames in all directions.

Realism

• Clue 1: A close-up shot focuses on a pair of calloused hands meticulously carving wood. The hands hold a carving knife, its blade gleaming amidst flying wood shavings, the wood grain clearly visible. Fingertips move lightly across the blade, outlining delicate patterns and designs. The background is blurred, showing only the workbench and scattered woodworking tools, emphasizing the exquisite movements of the hands and the essence of craftsmanship.

• Clue 2: A fixed shot from a 45-degree downward angle, medium shot, shallow depth of field, with dappled morning light filtering through the sycamore trees. In the center of the frame, an orange cat wearing an apron steadily slicing a cucumber, its tail swaying gently with the rhythm of the chopping. In the background, smoke rises from a stove, and sunlight casts a warm glow on the wooden counter and the old iron pot.

Character close-up

• Clue 1: A close-up of a woman: at first she is laughing, then she becomes sad, then she starts crying, and finally she covers her face with her hands.

Multi-person scenes/Large-scale scenes/Multi-camera shots

• Clue 1: Dawn, medieval style, two armies are engaged in a fierce battle.

Application scenarios of Wanxiang 2.1

• Film and television productionGenerate ancient war special effects for period dramas, such as thousands of troops charging and arrows raining down, to enhance the visual effects.
• Advertising video production: Create advertisements for sports drinks, generating scenarios of young people enjoying a refreshing drink after exercise, highlighting the product's features.
• Teaching aidsThe classroom generates a video of the Battle of Red Cliffs during history class to help students understand the historical event.
• Cultural CreationTo produce documentaries about calligraphy art, generate videos of calligraphers' creative process, and showcase calligraphy techniques.
• News reportReport traffic accidents and generate simulated accident scene videos to clearly show the course of the accident.