QianYing - A large-scale model for generating audio games launched by Giant Interactive.

What is QianYing?

QianYing is a large-scale audio game generation model launched by Giant Interactive, comprising YingGame (a large-scale game video generation model) and YingSound (a large-scale video dubbing model). YingGame, designed for open-world games, is a joint development of Giant Interactive's AI Lab and Tsinghua University's SATLab. It enables diverse character action interaction control and physics simulation, and when combined with YingSound, it can generate complex sound effects that match the visuals. YingSound, a joint development of Giant Interactive's AI Lab, Northwestern Polytechnical University's ASLP Lab, and Zhejiang University, adds realistic sound effects to silent videos and possesses video semantic understanding and time alignment capabilities. QianYing aims to lower the barrier to game development through AI technology, promote equal opportunities for game creation, and accelerate innovation through the integration of games and AI.

QianYing's main functions

• YingGame's main functions:
  • Character action interaction controlYingGame enables interactive control of various character actions, allowing users to customize the game character's movements and appearance.
  • Physical simulation characteristicsThe model possesses powerful physics simulation capabilities, capable of simulating real-world physical effects and enhancing the game's realism.
  • Video generationBy combining its self-developed video dubbing model YingSound, YingGame can generate complex sound effects that match the game visuals, thus enabling the generation of audio game videos.
  • Interactive capabilitiesYingGame can understand user input interactions, such as mouse and keyboard operations, and control characters to perform various actions.
• YingSound's main functions:
  • Temporal alignment and video semantic understandingYingSound boasts superior time alignment and video semantic understanding capabilities, enabling it to accurately add sound effects to videos.
  • High-precision sound generationThe model can generate various types of high-resolution sound effects for all kinds of videos, including real-world scenes, animation, and game videos.
  • Diverse application scenariosYingSound supports generating high-fidelity sound effects from videos or a combination of video and text, and has the ability to generalize to various application scenarios.
  • Cross-modal alignment effectBased on its innovative technical architecture and reinforcement learning, YingSound has achieved industry-leading levels in terms of generation quality, temporal alignment, and video semantic understanding.

The technical principles of QianYing

• YingGame's technical principles:
  • Physical simulation characteristicsThe integrated physics engine simulates the physical rules of the real world, such as gravity and collision, enhancing the realism and immersion of the game.
  • Cross-modal learningBased on cross-modal learning technology, visual information and sound effect information are combined to generate sound effects that match the game screen.
• YingSound's technical principles:
  • Audio-video synchronization technologyBased on precise time alignment technology, it ensures that the generated audio effects are synchronized with the video content, which involves complex time series analysis.
  • Deep Neural NetworksUsing deep neural networks to generate high-precision sound effects, the neural network can learn key features extracted from video content and convert them into corresponding sound effects.

QianYing's project address

• Project official website:ga-me.com

Application scenarios of QianYing

• Open World Game DevelopmentIn open-world games, create characters with complex movements and physical interactions to provide a more realistic and immersive gaming experience.
• Game video content creationGame developers generate high-quality game trailers, story animations, and other video content to enhance the game's market appeal.
• Game education and trainingIn the field of education, simulated environments and scenarios are created for use in skills training or safety education to improve learning efficiency and effectiveness.
• Game streaming and esportsAdd special effects and sound effects to game live streams to enhance their viewing experience and interactivity, while providing a more vivid viewing experience for esports events.
• Game testing and prototypingQuickly generate game prototypes, accelerate game design and testing processes, and reduce development time and costs.