Goku - The latest video generation model jointly launched by HKU and ByteDance

What is Goku?

Goku is a new video generation model jointly released by the University of Hong Kong and ByteDance, designed specifically for the joint generation of images and videos. Based on the advanced rectified flow Transformer framework, it supports various modes such as text-generated video, image-generated video, and text-generated image. Goku's core advantages include high-quality video generation capabilities and extremely low advertising video production costs (100 times lower than traditional methods). Goku is built upon a large-scale, high-quality dataset and efficient training facilities. Researchers constructed a dataset containing approximately 36 million videos and 160 million images, employing a multimodal large language model to generate context-consistent frameworks. Goku utilizes advanced parallel strategies and fault-tolerant mechanisms to ensure the efficiency and stability of the training process.

Goku+ is an extended version based on the Goku model, focusing on ad video creation. It can quickly generate high-quality ad videos, supporting videos longer than 20 seconds, and features stable hand gestures and rich facial and body expressions. Goku+ can transform product images into engaging videos, supporting virtual avatars interacting with products to enhance ad appeal. It is suitable for various scenarios such as e-commerce, brand promotion, short video ads, and product demonstrations, significantly reducing ad production costs and improving creative efficiency.

Goku's main functions

• Text-to-Image:It can generate high-quality images based on text descriptions, producing images that are rich in detail and highly consistent with the text descriptions.
• Text-to-VideoGenerates coherent videos based on text descriptions, featuring smooth motion and high-quality visuals.
• Image-to-VideoIt generates videos based on images, maintaining the visual style and semantic consistency of the images. It can generate dynamic videos from static images and is suitable for animation and video content creation.
• Ad video generation (Goku+)Generate high-quality advertising videos that support natural interaction between people and products. It can reduce advertising video production costs by 100 times, producing videos with stable hand movements and rich facial expressions.
• Virtual digital human video generationIt generates videos of virtual digital humans with a high degree of realism and natural movements, suitable for scenarios such as virtual anchors and virtual customer service.
• Multimodal generationIt supports multimodal generation tasks, including the joint generation of images, videos, and text. By sharing a latent space and employing a full attention mechanism, Goku can seamlessly handle complex spatiotemporal dependencies in images and videos.

Goku's technical principles

• Image-Video Joint VAEGoku employs a 3D joint image-video variational autoencoder (VAE) to compress image and video inputs into a shared latent space. This enables the model to handle multiple media formats, including images and videos, and represent them within a unified framework.
• Transformer architectureThe Goku model family includes Transformer architectures with 2B and 8B parameters. Based on a full attention mechanism, the architecture effectively handles complex spatiotemporal dependencies in images and videos, achieving high-quality, coherent output.
• Corrected flow formulaBased on the Rectified Flow (RF) algorithm, Goku trains between the prior distribution and the target data distribution using linear interpolation. Compared to traditional diffusion models, it exhibits faster convergence speed and stronger theoretical properties.
• Multi-stage training strategyGoku employs a multi-stage training strategy, including image-text semantic alignment pre-training, image-video joint training, and fine-tuning for different modalities. This progressively improves the model's generative capabilities, ensuring efficient training on large-scale datasets.
• Large-scale high-quality datasetsResearchers constructed a massive dataset of approximately 36 million videos and 160 million images, employing various data filtering and augmentation techniques to improve data quality. The dataset provides abundant material for model training.
• High-efficiency training infrastructureGoku's training infrastructure includes parallel strategies, fine-grained activation checkpointing, fault tolerance mechanisms, and ByteCheckpoint technology. These significantly improve training efficiency and stability.

Goku's project address

• Project official website:https://saiyan-world.github.io/goku/
• Github repository:https://github.com/Saiyan-World/goku
• HuggingFace model library:https://huggingface.co/datasets/saiyan-world/Goku
• arXiv technical paper:https://arxiv.org/pdf/2502.04896

Goku application scenarios

• Advertising video productionGoku+ can generate high-quality advertising videos based on text descriptions, supporting the generation of videos directly from text, the generation of interactive videos from product images, and the generation of product demonstration videos.
• Virtual digital human video generationGoku+ can convert text into surreal human videos, generating videos longer than 20 seconds with stable hand gestures and highly expressive facial and body movements.
• Content creationGoku can generate videos featuring a variety of scenes, including animation, natural landscapes, and animal behavior. It can generate videos of a stylish woman strolling through the streets of Tokyo, or scenes of several giant mammoths walking on the snow. This provides artists with a wealth of inspiration and creative material.
• Education and TrainingGoku can be used to create educational videos and training courses, enhancing the effectiveness and engagement of education and training by generating vivid video content.
• Entertainment industryIn the entertainment industry, including film, television, and animation, Goku can be used for content creation and special effects generation. It can generate high-quality video content, providing creators with more possibilities.