Tora - Alibaba's AI video generation framework
Tora is an AI video generation framework launched by Alibaba. Based on trajectory-guided diffusion transformer (DiT) technology, it fuses text, visual information, and trajectory conditions to generate high-quality video content that conforms to the dynamics of the physical world. Tora consists of trajectory...
What is Tora?
Tora is an AI video generation framework launched by Alibaba. Based on trajectory-guided diffusion transformer (DiT) technology, it fuses text, visual information, and trajectory conditions to generate high-quality video content that conforms to the dynamics of the physical world. Tora consists of a trajectory extractor, spatiotemporal DiT, and a motion-guided fusion processor, enabling precise control over the dynamic performance of videos and supporting video production up to 204 frames per second at 720p resolution. Tora excels in motion fidelity and simulation of real-world physical dynamics, providing a powerful new tool for the field of video generation.
Tora's main functions
Simply put, Tora can create realistic and smooth videos based on your instructions (such as text descriptions, images, or the path of an object's movement).
- Trajectory Extractor (TE)The input trajectory is transformed into hierarchical spatiotemporal motion blocks that match the potential space of the video content.
- Spatial-Temporal DiTBy combining spatial and temporal self-attention mechanisms to process video data, the model is able to understand and generate videos with coherent motion.
- Motion-guidance fuser (MGF): Responsible for integrating the spatiotemporal motion blocks generated by the trajectory extractor into the DiT model, ensuring that the generated video content follows the predetermined trajectory and dynamics.
Tora's technical principles
- Trajectory UnderstandingTora uses a tool called a "track extractor," which can understand given track information. It's like giving Tora a map and telling it where the objects in the video should be and how they should move.
- Spatiotemporal codingTora converts this trajectory information into a special encoded form called "spatiotemporal motion blocks." These motion blocks are like the skeleton of the video, determining how objects move in the video.
- Video generation frameworkTora employs an advanced technology called "Diffusion Transformer" (DiT). This technology combines the advantages of diffusion models and transformer architectures, enabling Tora to generate high-quality video.
- Dynamic fusionTora also has a "motion-guided fusion tool," which combines the previously obtained spatiotemporal motion blocks with the video content. Tora can ensure that the generated video not only looks good, but also that the movement of objects is very natural and smooth.
- Two-stage trainingTo enable Tora to better understand and generate motion, it learns through a two-stage training process. First, it learns how to extract motion information from dense optical flow (a type of dense data describing object motion). Second, it learns how to generate video based on simpler trajectory information provided by the user.
- Data preprocessingBefore training, Tora also needs to process the video data, such as segmenting long videos into short segments based on scene detection, and then selecting suitable video segments for training based on aesthetic scores and motion segmentation results.
Tora's project address
- Project official website:https://ali-videoai.github.io/tora_video/
- GitHub repository:https://github.com/ali-videoai/Tora
- arXiv technical paper:https://arxiv.org/pdf/2407.21705
Tora's application scenarios
- Film and television productionTora can be used to generate special effects scenes in movies, TV series, or short films. It generates complex dynamic images through trajectory control, reducing actual shooting costs and time.
- Animation CreationIn the field of animation, Tora can automatically generate animation sequences based on scripts, providing animators with initial dynamic sketches and accelerating the creative process.
- Virtual Reality (VR) and Augmented Reality (AR)Tora can generate dynamic environments that interact with users, providing realistic visuals for VR and AR applications.
- Game developmentIn video games, Tora can be used to quickly generate game environments and character animations, improving the efficiency of game design.