Ruyi - A large-scale video model launched by TuSimple.

What is Ruyi?

Ruyi is a large-scale video generation platform launched by TuSimple, designed specifically for running on consumer-grade graphics cards. It supports multi-resolution and multi-duration video generation, and features first-frame and first-and-last-frame control, motion amplitude control, and camera control. Based on the DiT architecture, Ruyi consists of a Casual VAE module and a Diffusion Transformer for video data compression and generation. Ruyi can reduce the development cycle and cost of animation and game content, making it an ideal tool for ACG enthusiasts and creators. TuSimple has now officially open-sourced the Ruyi-Mini-7B version.

Ruyi's main functions

• Multi-resolution, multi-duration generationRuyi supports video generation with resolutions ranging from a minimum of 384×384 to a maximum of 1024×1024, can handle any aspect ratio, and can generate videos up to 120 frames per second for 5 seconds.
• First frame and first and last frame control generationGenerates a video based on a maximum of 5 start frames and a maximum of 5 end frames, and generates a video of arbitrary length by cyclic superposition.
• Motion amplitude controlIt offers four levels of motion amplitude control, allowing users to easily control the degree of change in the overall image.
• Lens controlIt provides five camera controls: top, bottom, left, right, and still, increasing the flexibility of video generation.

Ruyi's technical principles

• Model ArchitectureRuyi is based on the DiT (Diffusion Model with Transformers) architecture and consists of two parts:
  • Casual VAE module: Responsible for compressing and decompressing video data.
  • Diffusion Transformer: Responsible for generating the compressed video.
• Compression and EncodingThe Casual VAE module compresses the spatial resolution to 1/8 and the temporal resolution to 1/4. After compression, each pixel is represented by a 16-bit BF16.
• Location codingThe DiT part uses 3D full attention, spatially using 2D RoPE (Rotary Positional Encoding) for positional encoding, and temporally using sin_cos for positional encoding.
• Training loss functionThe final loss function was trained using DDPM (Denoising Diffusion Probabilistic Models).
• Number of parameters and training dataThe model has a total of approximately 7.1 bytes of parameters and was trained using approximately 200 million video clips.
• Training phaseThe entire training process is divided into four stages, from low-resolution pre-training to high-resolution fine-tuning, gradually improving model performance.

Ruyi's project address

• GitHub repository:https://github.com/IamCreateAI/Ruyi-Models
• HuggingFace model library:https://huggingface.co/IamCreateAI/Ruyi-Mini-7B

Ruyi's application scenarios

• Animation pre-madeQuickly generate dynamic demos of animated characters and scenes, and evaluate animation design and story flow.
• Game CG generationIn game development, it can automatically generate game promotional videos or dynamic backgrounds within the game.
• Movie special effects previewIn film production, it helps directors and special effects teams preview special effects scenes and optimize post-production special effects.
• Virtual streamerGenerate dynamic videos of virtual anchors for use in live streaming, news broadcasting, or online education.
• Social media contentCreate engaging and dynamic video content for social media platforms to increase user engagement and brand exposure.