Seaweed APT - ByteDance's one-step image and video generation project

What is Seaweed APT?

Seaweed APT, developed by ByteDance, is an adversarial post-training model capable of one-stop image and video generation. Based on a pre-trained diffusion model, Seaweed APT directly trains against real data in an adversarial manner, rather than using the pre-trained model as a teacher to generate targets, achieving high-quality image and video output in a single generation step. Seaweed APT introduces several improvements, including a generator initialized with deterministic distillation, an enhanced discriminator architecture, and an approximate R1 regularization objective, improving training stability and generation quality. Experiments show that Seaweed APT can generate 1024px images and 2-second, 1280×720, 24fps videos in a single step, surpassing traditional multi-step diffusion models in visual fidelity.

Main functions of Seaweed APT

• Single-step high-quality image generationIt can generate images with a resolution of 1024px, and is comparable to the multi-step diffusion model in terms of visual fidelity, detail and structural integrity.
• Single-step high-resolution video generationFor the first time, it achieves 1280×720, 24fps, 2-second video generation, significantly improving the efficiency and quality of video generation.
• Real-time generation capabilityOn a single H100 GPU, Seaweed APT can generate a 2-second 1280×720 24fps video in 6.03 seconds, based on parallelization of 8 H100 GPUs, with the entire generation process being processed in real time.
• Beyond pre-trained modelsBy performing adversarial training directly on real data, Seaweed APT avoids the high cost of pre-computed video samples and outperforms pre-trained diffusion models on certain evaluation metrics, particularly in terms of realism, detail, and exposure.

Seaweed APT Technical Principles

• Pre-trained diffusion model initializationUsing a pre-trained Diffusion Transformer (DiT) as initialization avoids the high cost of pre-computing video samples and outperforms pre-trained models on some evaluation metrics.
• competitive trainingTraining on real data based on adversarial training objectives generates samples that are closer to the real distribution, improving the quality and realism of the generated samples.
• Generator DesignThe generator is initialized using deterministic distillation, employing discrete-time consistency distillation and mean squared error loss. During training, the generator primarily focuses on single-step generation capability, consistently inputting the final time step T.
• Discriminator DesignThe discriminator is initialized using a pre-trained diffusion network and operates directly in the latent space. The discriminator contains 36 Transformer layers with a total of 8 billion parameters. By introducing new cross-attention blocks at multiple time steps, the discriminator can better distinguish between real and generated samples.
• Approximate R1 regularizationTo address the problem of calculating higher-order gradients for R1 regularization in large-scale Transformer models, an approximation method is proposed. This method adds small-variance Gaussian noise to the real data, reducing the gradient of the discriminator on the real data and achieving the same goal as the original R1 regularization.
• Training detailsThe training process begins with images and then moves to videos. A large batch size is achieved using numerous H100 GPUs and gradient accumulation, improving training stability and structural integrity. An exponential moving average (EMA) decay rate is employed to stabilize the training process, and the learning rate is adjusted during training to ensure stability.

Seaweed APT project address

• Project official website:https://seaweed-apt.com
• arXiv technical paper:https://arxiv.org/pdf/2501.08316

Application scenarios of Seaweed APT

• Video Ad ProductionQuickly generate high-quality, diverse video ads, reducing production costs and time.
• Film and television content creationGenerate video clips with specific styles for movies and TV series, improving creative efficiency and content diversity.
• Social media contentGenerate personalized, high-quality videos and images to enhance user engagement and content appeal.
• Game developmentQuickly generate images of characters, scenes, and props in games, accelerating the game development process.
• Education and TrainingGenerate educational videos and training materials to improve teaching effectiveness and training quality.