DiffuEraser - A video restoration model developed by Ali Tongyi Labs

What is DiffuEraser?

DiffuEraser is a video inpainting model based on a stable diffusion model, filling masked regions in videos with richer details and more coherent structure. The model provides initialization and weak conditions by incorporating prior information, which helps reduce noise artifacts and suppress hallucinations. To improve temporal consistency during long-sequence inference, DiffuEraser extends the temporal receptive field of the prior model and its own, further enhancing consistency based on the temporal smoothing properties of the video diffusion model. Inspired by AnimateDiff, DiffuEraser's network architecture integrates a motion module into the image inpainting model. It mainly consists of a main denoising UNet and an auxiliary BrushNet. The BrushNet branch receives conditional latent variable input consisting of the masked image, the mask, and noise latent variables. Features extracted by BrushNet are layer-by-layer integrated into the denoising UNet after passing through zero-convolutional blocks. The denoising UNet processes the noise latent variables. To enhance temporal consistency, a temporal attention mechanism is introduced after self-attention and cross-attention layers. After denoising, the generated image is fused with the input masked image using a blurred mask.

DiffuEraser's main functions

• Generation of unknown pixelsBased on the powerful generation capabilities of the stable diffusion model, DiffuEraser can generate reasonable content with rich details and textures for pixels that have never appeared in the video before, effectively solving the blurring and mosaic problems that are common in traditional Transformer models when dealing with large masks.
• Propagation of known pixelsDiffuEraser enhances propagation capabilities through motion modules and prior models, ensuring that known pixels (pixels that have appeared in some masked frames) are propagated sufficiently and consistently across different frames. This prevents conflicts between the repaired content and the unmasked areas, improving the accuracy and stability of the results.
• Time consistency maintenanceDuring long sequence inference, DiffuEraser enhances the temporal consistency of completed content across all frames by expanding the prior model and its own temporal receptive field, based on the temporal smoothing properties of the video diffusion model.
• Injection of prior informationDiffuEraser provides initialization and weak conditions by injecting prior information, which helps reduce noise artifacts, suppress visual illusions common in diffusion models, and generate more accurate and realistic repair results.
• Network architecture optimizationThe network architecture of DiffuEraser is inspired by AnimateDiff. It integrates a motion module into the image inpainting model BrushNet and further enhances temporal consistency by introducing a temporal attention mechanism after self-attention and cross-attention layers.

DiffuEraser's technical principles

• Network architectureInspired by AnimateDiff, DiffuEraser's network architecture integrates a motion module into the image inpainting model BrushNet. The architecture primarily consists of a main denoising UNet and an auxiliary BrushNet. The BrushNet branch receives conditional latent variable input consisting of a mask image, the mask itself, and noise latent variables. The extracted features are then layer-by-layer integrated into the denoising UNet after passing through zero-convolutional blocks. The denoising UNet handles the noise latent variables. To enhance temporal consistency, a temporal attention mechanism is introduced after self-attention and cross-attention layers. After denoising, the generated image is fused with the input mask image using a blurred mask.
• Breakdown of video repair issuesDiffuEraser breaks down the video restoration problem into three sub-problems: propagation of known pixels, generation of unknown pixels, and maintenance of temporal consistency of the completed content.
• Time consistency optimizationTo improve temporal consistency during long sequence inference, DiffuEraser employs the following strategies:
  • Extending the temporal receptive field of the prior modelThrough pre-propagation and pre-inference, the prior model can propagate known pixels across the entire time domain, ensuring consistency between the completed content and the unmasked area, and stabilizing the results.
  • Expanding the temporal receptive field of DiffuEraserBy using pre-inference, DiffuEraser is able to process a wider range of video frames, ensuring consistency in the content generated during long-sequence inference.

DiffuEraser project address

• Project official website:https://lixiaowen-xw.github.io/DiffuEraser-page/
• GitHub repository:https://github.com/lixiaowen-xw/DiffuEraser
• arXiv technical paper:https://arxiv.org/pdf/2501.10018

Application scenarios of DiffuEraser

• Post-production of movies and TV seriesIn the post-production of movies or TV series, DiffuEraser can be used to repair masked areas in videos, improve video quality, perform deblurring and super-resolution processing, and adapt to playback requirements of different resolutions.
• Old film restorationFor the digital restoration of old films, DiffuEraser can remove scratches, dust and other degradation from the film, improve resolution, and give old movies a new lease on life.
• Enhanced surveillance videoIn the field of security monitoring, DiffuEraser can enhance the clarity of surveillance videos, help identify details, and improve monitoring efficiency.
• Video content conversionContent creators can use DiffuEraser to convert standard definition (SD) video content to high definition (HD) or 4K to meet the needs of modern display devices.
• Live sports broadcastsIn live sports broadcasts, DiffuEraser can be used to enhance real-time video streams, providing a clearer viewing experience.