AnimeGamer - A simulation system for anime life developed by Tencent in collaboration with City University of Hong Kong.
AnimeGamer is an infinite anime life simulation system jointly developed by Tencent PCG and City University of Hong Kong. Based on a multimodal large language model (MLLM), it allows players to immerse themselves in anime life using open-ended language commands, embodying the characters of anime...
What is AnimeGamer?
AnimeGamer is an infinite anime life simulation system jointly developed by Tencent PCG and City University of Hong Kong. Based on a multimodal large language model (MLLM), it allows players to immerse themselves in a dynamic game world as an anime character using open-ended language commands. Players can control characters such as Sousuke from *Ponyo* and interact with the game world. The game supports the generation of context-consistent dynamic animation shots (videos) and updates to character states (such as stamina, social, and entertainment values). Compared to traditional methods, AnimeGamer excels in character consistency, semantic consistency, and motion control, providing players with an immersive anime game experience.
Main functions of AnimeGame
- Role-playing and interactionPlayers take on the role of anime characters, such as Sousuke from "Ponyo on the Cliff by the Sea," and interact with the game world, allowing characters from different anime to meet and interact with each other.
- Dynamic animation generationBased on player commands, it generates dynamic animated shots (videos) in real time, showcasing character actions and scene changes, and the animations are context-consistent and dynamic.
- Character status updateBased on character behavior and interactions, dynamically update the character's stamina, social, and entertainment values to reflect changes in the character's status in the game world.
- Multi-turn dialogue interactionIt supports multi-turn dialogues between players based on natural language, and the model generates a consistent game state based on historical context, providing a coherent gaming experience.
- Customize game contentIt supports players to customize their favorite characters and scenes.
The technical principles of AnimeGame
- Multimodal large language modelAnimeGamer uses a multimodal large language model (MLLM) to understand and generate multimodal data containing textual and visual information.
- Action-aware multimodal representationThe game breaks down animated shots into three parts: visual reference, motion description, and motion intensity. An encoder integrates this information into a multimodal representation. A video diffusion model is used to decode the animation into high-quality dynamic video, ensuring the generated animated shots possess contextual consistency and dynamism.
- Video diffusion modelVideo diffusion models (such as CogVideoX) act as decoders for animated shots, decoding multimodal representations into dynamic video. Introducing motion intensity as an additional condition controls the amplitude of motion in the generated video, making the animation more natural and realistic.
- Context consistencyBy using multimodal representations of historical animation shots as contextual input, the model can predict subsequent game states and ensure that the generated animation shots remain consistent within the context, which is crucial for maintaining the game's coherence and immersion.
- Role status managementThe game uses MLLM (Multi-Level Modeling) to predict and update a character's stamina, social, and entertainment stats. These status updates reflect the character's behavior and interactions in the game world. This dynamic updating of character status enhances the game's realism and interactivity.
AnimeGame's project address
- Project official website:https://howe125.github.io/AnimeGamer.github.io/
- GitHub repository:https://github.com/TencentARC/AnimeGamer
- HuggingFace model library:https://huggingface.co/TencentARC/AnimeGamer
- arXiv technical paper:https://arxiv.org/pdf/2504.01014
Application scenarios of AnimeGame
- Personalized entertainmentPlayers can choose their favorite anime characters and scenes, and experience a unique adventure story based on language commands.
- Creative inspirationIt provides inspiration for creators, generating character interactions and new storylines.
- Educational SupportIt helps students learn language expression and logical thinking.
- Social interactionPlayers can create and share anime adventure stories with their friends.
- Game developmentIt helps developers quickly generate game content and reduce development costs.