GenMAC - HKU, Tsinghua University, and Microsoft jointly launch a multi-agent collaboration framework for text-to-video generation.
GenMAC is an iterative framework for multi-agent collaboration developed by the University of Hong Kong, Tsinghua University, and Microsoft Research, used to solve complex scene generation problems in text-to-video conversion. It is based on breaking down the task into design, generation, and redesign...
What is GenMAC?
GenMAC is an iterative framework for multi-agent collaboration, developed by the University of Hong Kong, Tsinghua University, and Microsoft Research, designed to solve complex scene generation problems in text-to-video conversion. It decomposes the task into three phases: design, generation, and redesign, establishing an iterative loop between generation and redesign to progressively verify and optimize video content. The redesign phase is further subdivided into four subtasks: verification, suggestion, correction, and structured output. These are executed sequentially by specialized agents, using an adaptive self-routing mechanism to select the appropriate agent for the current scene, achieving more accurate video generation.
Main functions of GenMAC
- Combined text to video generationIt generates videos based on complex combined text prompts, handling scenarios involving multiple objects, attribute binding, time dynamics, and interactions between objects.
- Iterative WorkflowThe video content is gradually improved using an iterative approach, which includes three stages: design, generation, and redesign, and the iterative cycle between them.
- Multi-agent collaborationThe framework uses multiple specialized MLLM (Multimodal Large Language Model) agents, each responsible for handling specific sub-tasks, to achieve collective intelligence.
- Task breakdownThe redesign phase is broken down into four subtasks: verification, recommendation, correction, and output structure, which are executed in a different agent sequence.
- Adaptive self-routing mechanismGenMAC employs an adaptive self-routing mechanism to adapt to different generation scenarios, selecting the most suitable proxy from a dedicated set of proxies for correction.
- Improve scene accuracy and text alignment: Improve the accuracy of video scenes and alignment with text prompts by using multi-agent collaboration and iterative refinement.
GenMAC's technical principles
- Task decomposition and role specializationThe complex video generation task is broken down into simpler subtasks, and a dedicated agent is assigned to each subtask, with each agent having a specific role and responsibility.
- Iteration loopAn iterative loop is set up between the generation and redesign phases, allowing the model to progressively validate and correct the generated video, better aligning with text prompts.
- Agent collaboration:
- Authentication ProxyCheck the alignment between the video content and the text prompts.
- Recommended AgentBased on the verification results, propose correction suggestions and select a suitable correction agent.
- Modify proxyAdjust the video design based on the suggestions, such as layout and guideline proportions.
- Output structured proxyThe correction results are converted into a structured format to provide input for the next iteration.
- Adaptive self-routingBased on different generation requirements and scenarios, it adaptively selects the most suitable correction agent to handle issues such as consistency, temporal dynamics, and spatial dynamics.
- Cross-stage information flowInformation such as layout, guide proportions, and text prompts is continuously updated and passed between the design, generation, and redesign phases to achieve more accurate video generation.
GenMAC's project address
- Project official website:karine-h.github.io/GenMAC
- GitHub repository:https://github.com/Karine-Huang/GenMAC
- arXiv technical paper:https://arxiv.org/pdf/2412.04440
GenMAC application scenarios
- Film and video productionGenerate video clips based on the text descriptions in the script or storyboard to help directors and producers preview scenes.
- Game developmentIn game design, it generates concept videos of game environments and dynamic scenes to assist game designers in their creative process.
- Advertising and MarketingGenerate video ads based on advertising copy, quickly transforming creative copy into visual content and improving the efficiency of ad production.
- Education and trainingCreate educational videos to present complex theories or historical events in video format, enhancing the learning experience.
- News and MediaAutomatically generate news videos based on press releases or reports, improving the efficiency and responsiveness of news production.