Mora - A multi-AI agent framework developed by Microsoft and others that can generate 12-second videos.

What is Mora?

Mora is a multi-agent (AI agent) framework developed by researchers from Microsoft and Lehigh University, specifically designed for general video generation tasks. Its goal is to simulate and extend OpenAI's Sora video generation model. The core idea of this framework is to generate high-quality video content through the collaboration of multiple visual agents. Mora decomposes the video generation process into multiple sub-tasks and assigns a dedicated agent to each sub-task, thereby enabling various video generation functionalities.

According to the experimental results in the paper, Mora performs excellently in generating high-resolution (1024×576) videos lasting 12 seconds, containing a total of 75 frames. However, when dealing with scenes involving a large number of moving objects, Mora shows a significant performance gap compared to Sora. Furthermore, attempting to generate videos longer than 12 seconds results in a significant degradation in video quality.

Mora's main functions

• Text to video generationMora can automatically generate corresponding video content based on the text descriptions provided by users, and can be used for video creation ranging from simple scene descriptions to complex storylines.
• Image to video generationIn addition to generating videos directly from text, Mora can also combine initial images and text prompts provided by the user to generate matching video sequences, enhancing the richness and detail of the content.
• Extended video generationMora can not only generate videos from scratch, but also expand and edit existing video content, adding new elements or extending the duration of the video.
• Video to video editingMora features advanced editing capabilities, allowing users to edit videos based on text commands, such as changing scenes, adjusting object properties, or adding new elements.
• Connect videoMora can seamlessly connect two or more video clips to create smooth transitions, making it suitable for creating video compilations or edits.
• Analog Digital WorldMora can also create and simulate digital worlds, generating video sequences with a digital world style based on text descriptions, such as game scenes or virtual environments.

Mora's official website entrance

• GitHub address:https://github.com/lichao-sun/Mora(Source code and model to be open source)
• arXiv research paper:http://arxiv.org/abs/2403.13248

How Mora works

Mora works based on a multi-agent framework that coordinates multiple specialized AI agents to complete video generation tasks. Each agent is responsible for handling specific sub-tasks, which together constitute the complete video generation process.

The following are the detailed steps of the Mora workflow:

1. Task breakdownMora breaks down complex video generation tasks into multiple subtasks, each handled by a dedicated agent.
2. Intelligent agent role definitionMora defined the following five basic roles of intelligent agents:
   • Prompt for selecting and generating intelligent agentsUse large language models (such as GPT-4 or Llama) to optimize and select text prompts to improve the relevance and quality of generated images.
   • Text-to-image generation intelligent agent: Convert text prompts into high-quality initial images.
   • Image-to-image generation intelligent agentModify the given source image according to the text instructions.
   • Image-to-video generation intelligent agentConvert static images into dynamic video sequences.
   • Video-connected intelligent agentsCreate a video with a smooth transition based on two input videos.
3. WorkflowMora automatically organizes agents to execute subtasks in a specific order based on task requirements. For example, a text-to-video generation task might include the following steps:
   • first,Prompt for selecting and generating intelligent agentsHandle text prompts.
   • then,Text-to-image generation intelligent agentGenerate an initial image based on the optimized text prompts.
   • Then,Image-to-video generation intelligent agentConvert the initial image into a video sequence.
   • Finally, if necessary.Video-connected intelligent agentsMultiple video clips can be linked together to form a coherent video.
4. Multi-agent collaborationThe agents communicate and collaborate through predefined interfaces and protocols to ensure the coherence and consistency of the entire video generation process.
5. Generation and EvaluationEach agent, after completing its subtask, passes the result to the next agent until the entire video generation process is complete. The generated video can be quality-assessed based on predefined evaluation criteria.
6. Iteration and optimizationThe Mora framework allows for improvements in video generation quality through iteration and optimization. The agent can adjust its parameters based on feedback to enhance the quality of the generated video and its consistency with text prompts.