AB
AB
AiBoss

All Categories

project

Movie Gen - Meta launches text-driven AI video generation and editing tool

Movie Gen is an AI video generation tool launched by Meta. It can generate and edit videos based on text prompts and add synchronized audio. The technology includes creating high-definition videos up to 16 seconds long, adding audio to existing videos, editing...

What is Movie Gen?

Movie Gen is an AI video generation tool launched by Meta that can generate and edit videos based on text prompts, and add synchronized audio. Its technology includes creating high-definition videos up to 16 seconds long, adding audio to existing videos, editing videos, and creating custom videos based on photos. Movie Gen's video generation model has 30-bit parameters and can generate videos at 16 frames per second. Its audio generation model has 13-bit parameters and can generate high-quality audio up to 45 seconds long, including ambient sounds, sound effects, and instrumental background music. The tool is currently only available to internal employees and a limited number of partners. Meta plans to integrate it into existing applications in the future to increase user engagement.

Movie Gen's main functions

  • Video generationThe user inputs text prompts, and Movie Gen generates a 16-second high-definition video. The video is automatically created based on the text description and adapts to different aspect ratios and resolutions.
  • Personalized video generationUsers upload photos, and Movie Gen can generate videos that include the characteristics of that person while maintaining consistency in the person's identity and actions.
  • Precision video editingMovie Gen allows users to precisely edit videos using text commands, including adding, deleting, or replacing elements in the video, as well as changing the background or style.
  • Audio generationMovie Gen generates high-quality, synchronized audio for videos, including ambient sounds, foley sounds, and instrumental background music. The audio is matched to the video content and generated based on text prompts.

The technical principles of Movie Gen

  • Model ArchitectureMovie Gen includes two main base models: Movie Gen Video (video generation) and Movie Gen Audio (audio generation).
  • Pre-trainingThe model is pre-trained on a large amount of internet image, video and audio data to learn various concepts of the visual and auditory world.
  • Flow Matching: Train a generative model using stream matching to train the target, a method for iteratively updating samples, used in generating data.
  • Variational Autoencoder (VAE)Variational autoencoders are used to compress videos and images into a learned latent space, improving training and inference efficiency.
  • Text embeddingThe input text is converted into text embeddings using a pre-trained text encoder, which serve as conditional inputs to the model.
  • Multimodal inputThe model can handle various input types, including text, images, and videos, and generate corresponding outputs.

Movie Gen's project address

Application scenarios of Movie Gen

  • Content creationCreate unique video content for social media, blogs, or video sharing platforms.
  • Film and video production: Generate concept videos during the pre-production stage, or perform rapid editing and audio mixing during post-production.
  • Advertising and MarketingQuickly generate engaging ad videos to capture the attention of potential customers.
  • Education and trainingCreate educational content, such as simulations, historical reenactments, or language learning materials.
  • Game developmentGenerate video and audio for in-game video content, trailers, or promotional materials.
  • News and reportsQuickly generate visual content for news stories to enhance the appeal of your reporting.
Back