GameFactory - A generalizable game scenario framework jointly launched by the University of Hong Kong and Kuaishou.

What is GameFactory?

GameFactory is an innovative framework jointly proposed by the University of Hong Kong and Kuaishou Technology to solve the problem of scene generalization in game video generation. Based on a pre-trained video diffusion model, the framework combines open-domain video data and small-scale, high-quality game datasets, employing a multi-stage training strategy to achieve controllable game video generation.

GameFactory's main functions

• Scene generalization abilityGameFactory can generate diverse game scenes, not limited to a single style or preset scenes, making the generated videos more realistic and varied.
• Action controllabilityThrough the motion control module, GameFactory can achieve precise control over the movements of characters or objects in the generated video.
• High-quality dataset supportTo support motion-controlled video generation, GameFactory used the GF-Minecraft dataset. This dataset contains 70 hours of Minecraft gameplay videos with diverse scenes and detailed motion annotations.
• Interactive video generationGameFactory supports generating interactive game videos of unlimited length. Users can control changes to the video content by inputting commands or interacting with it.

GameFactory's technical principles

• Pre-trained video diffusion modelGameFactory, based on a video diffusion model pre-trained on open-domain video data, can generate diverse game scenes, breaking through the limitations of traditional methods on specific game styles and scenes.
• Three-stage training strategy
  • Phase 1We use LoRA (Low-Rank Adaptation) to fine-tune the pre-trained model to fit the target game domain while retaining most of the original parameters. This ensures that the model's generalization ability in open domains is preserved.
  • Phase TwoThis approach freezes pre-training parameters and LoRA, focusing on training the motion control module to avoid entanglement between style and control signals. The goal is to ensure that the generated video can accurately control motion based on user input.
  • Phase ThreeRemove LoRA weights while retaining motion control module parameters, enabling the system to generate controlled game videos in various open-domain scenarios, without being limited to a specific game style.
• Motion control moduleGameFactory introduces a motion control module that supports autoregressive motion control and can generate interactive game videos of unlimited length.
• GF-Minecraft DatasetTo support the generation of motion-controlled videos, the research team released the high-quality motion-annotated video dataset GF-Minecraft. The dataset contains 70 hours of Minecraft gameplay videos, featuring diverse scenes and detailed motion annotations. The dataset was designed to meet the following three key requirements:
  • Customizable actionsSupports large-scale, low-cost data collection.
  • Unbiased action sequence: Ensure the diversity of action combinations and coverage of low-probability events.
  • Diverse scenariosEnhance scene diversity by pre-configuring environmental settings (such as different biological communities, weather conditions, and time periods).

GameFactory's project address

• Project official website:https://vvictoryuki.github.io/gamefactory
• arXiv technical paper:https://arxiv.org/pdf/2501.08325

Application scenarios of GameFactory

• Game developmentGameFactory uses a pre-trained video diffusion model and motion control module to generate diverse game scenes and interactive video content.
• Game content innovationGameFactory's open-domain generation capabilities enable it to break through the limitations of traditional game engines and create entirely new game styles and scenarios.
• autonomous drivingIts motion control module and scene generation capabilities can be used to simulate autonomous driving environments and generate diverse driving scenarios.
• Embodied AIIt supports research on embodied intelligence by generating diverse interactive scenarios.