Pippo - Meta's single-image model for generating multi-view high-definition portrait videos

What is Pippo?

Pippo, an image-to-video generation model from Meta Reality Labs, can generate 1K resolution multi-view high-definition portrait videos from a single photograph. The model is based on a multi-view diffusion transformer, pre-trained on 3 billion portrait images, and post-trained on 2,500 studio-captured images. Pippo's core technologies include a ControlMLP module for injecting pixel alignment conditions, and attention bias techniques that enable it to generate more than 5 times more views during inference than during training. Pippo introduces reprojection error to evaluate the 3D consistency of multi-view generation.

Pippo's main functions

• Multi-view generationPippo can generate high-definition videos from multiple perspectives from a single full-body or facial photo, supporting the generation of full-body, facial, or head-only videos.
• High-efficiency content generationPippo can generate up to 5 times more video content than the training viewpoint using a multi-view diffusion transformer.
• High resolution supportPippo has achieved consistent multi-view portrait generation at 1K resolution for the first time.
• Spatial Anchor Points and ControlMLPBetter 3D consistency is achieved by injecting pixel alignment conditions, such as Plücker rays and spatial anchors, through the ControlMLP module.
• Automatic completion detailsWhen processing monocular video, Pippo can automatically fill in missing details, such as shoes, face, or neck.

Pippo's technical principles

• Multi-stage training strategy:
  • Pre-training phasePippo was first pre-trained on 3 billion unstructured human portrait images without any annotation information.
  • Intermediate training phaseOn a high-quality studio dataset, the model jointly generates images from multiple perspectives, uses low-resolution views for denoising, and coarsely encodes the target camera through a shallow MLP.
  • Post-training phase: Denoise a small number of views at high resolution and introduce pixel alignment controls (such as spatial anchors and Plücker rays) to achieve 3D consistency.
• Pixel alignment control (ControlMLP module)Pippo injects pixel alignment conditions, such as Plücker rays and spatial anchors, through the ControlMLP module. These conditions are used during the training phase and fixed to arbitrary positions during the inference phase.
• Attention bias technologyDuring the inference phase, Pippo proposed an attention bias technique that allows the model to generate more than five times more viewpoints simultaneously than during the training phase.
• 3D Consistency Assessment IndicatorsPippo introduces an improved 3D consistency evaluation metric—Re-projection Error—to assess the consistency of 3D generated from multiple viewpoints.

Pippo's project address

• Project official website:https://yashkant.github.io/pippo/
• Github repository:https://github.com/facebookresearch/pippo
• Technical Papers:https://yashkant.github.io/pippo/pippo.pdf

Pippo Application Scenarios

• Virtual Reality (VR) and Augmented Reality (AR)Pippo can generate high-quality multi-view portrait videos, which can be directly used for virtual character generation in VR and AR environments to enhance immersion.
• Film and television productionIn film and television post-production, Pippo can be used to quickly generate multi-view special effects shots, reducing shooting costs and time.
• videoconferenceBy generating multi-view videos from a single photo, Pippo can provide a more natural interactive experience for remote participants in video conferencing.
• Game developmentPippo can be used to generate character animations in games, enhancing the visual effects and player immersion.
• Social media and content creationCreators can use Pippo to generate multi-view videos, providing richer content for social media platforms.