MinT - A text-to-video model for generating sequential events, with flexible control over timestamps.

What is MinT?

MinT (Mind the Time) is a multi-event video generation framework jointly developed by Snap Research, the University of Toronto, and the Vector Institute. Based on precise time control, it generates video sequences containing multiple events according to text prompts. MinT's core technology is ReRoPE (Reliable Time-Based Position Encoding), which allows the model to associate specific text prompts with corresponding time segments in the video, ensuring events occur sequentially and controlling the duration of each event. As the first model to provide time control for events in video, MinT surpasses existing open-source models in generating coherent, dynamically connected events, bringing new flexibility and control to video content creation.

MinT's main functions

• Multi-event video generationGenerates a video containing multiple events based on text prompts. Events can be actions, expressions, or daily activities.
• Time controlIt allows users to specify the start and end times of each event, enabling precise control over the order and duration of events in the video.
• Maintaining coherenceWhen generating multi-event videos, maintain the coherence of the video content, including a consistent theme and background.
• High-quality video compositing: Optimize the model to generate high-quality video content, ensuring the video's dynamism and visual quality.
• LLM-based prompt enhancerIt provides a cue enhancer based on a large language model (LLM), which can expand short cues into detailed global and temporal captions, generating richer video content.

MinT's technical principles

• Time-Based Location Encoding (ReRoPE)eRoPE, a temporal-based location coding method, is introduced to guide the model on how to associate text cues with video frames, ensuring that events occur within the correct time frame.
• Pre-trained Video Diffusion Transformer (DiT)Based on a pre-trained Latent Diffusion Transformer (DiT), the transformer encodes the video into video tokens based on the encoder, and then generates the video using a denoising network.
• Cross-attention between global and time captionsTwo cross-attention layers are used in each DiT block to handle global captions and time captions respectively, enabling precise control over events.
• Relationship of event time rangeProvides time-based captions; the model focuses on one event at a time, maintaining control over the event's time range during the generation process.
• Scene switching controlBased on conditional modeling, scene transitions in videos are identified, and the effects of camera transitions are controlled in the generated videos.
• Enhanced promptsExtend short text prompts with a large language model (LLM) to generate detailed global and time captions, enabling users to generate complex video content from simple prompts.

MinT's project address

• Project official website:mint-video.github.io
• arXiv technical paper:https://arxiv.org/pdf/2412.05263v1

Application scenarios of MinT

• Entertainment industryUsed in the pre-production stage of movies and TV series to generate dynamic previews of the script, helping the team understand the flow of scenes and events.
• Advertising industryCreate dynamic ads to enhance the effectiveness of information delivery by precisely controlling the timing of product display in the ad.
• Education: Create educational videos, such as historical reenactments and scientific experiment simulations, to visually aid in learning and understanding complex concepts.
• gaming industryGenerate in-game story animations and cutscenes to enhance the game's narrative experience.
• News mediaQuickly create dynamic summary videos of news events to enhance the appeal of news reports and improve information delivery efficiency.