ProPainter - An AI video restoration project launched by Nanyang Technological University

What is ProPainter?

ProPainter is an AI video restoration project developed by the S-Lab team at Nanyang Technological University. By combining dual-domain propagation and mask-guided sparse video Transformer, it effectively improves the performance of video restoration. ProPainter can automatically remove unwanted objects from videos, fill in missing parts, and even expand the video's field of view, providing high-quality visual content. It is suitable for fields such as film post-production, historical video restoration, and social media content creation.

ProPainter's main functions

• Object removalIt can automatically detect and remove unwanted objects from videos, such as billboards and pedestrians, while maintaining the continuity and naturalness of the background.
• Video completionProPainter can intelligently predict and fill in any damaged or missing parts during video capture or transmission, restoring video integrity and smoothness.
• Video expansionTechnology can expand the existing field of view of videos by intelligently generating new backgrounds and scenes based on video content, providing a broader visual experience.

ProPainter's technical principles

• Dual-domain propagation:It combines the propagation advantages of the image domain and the feature domain, and utilizes global correspondence to achieve reliable information propagation.Image propagation uses an optical flow-based deformation alignment technique for global pixel-level propagation.Feature propagation uses learned offsets to refine propagation in the feature domain, improving robustness to occlusion and inaccurate optical flow.
• Mask-guided sparse video Transformer:Sparse attention mechanisms are used to reduce computational complexity and memory usage by discarding unnecessary and redundant windows in the query and key/value space.Attention mechanisms are applied only to query windows that intersect with the masked region, and time steps are used in the key/value space to selectively include frames.
• Circulating optical flow completion network:An efficient cyclic network is used to complete the damaged optical flow field, providing accurate optical flow information for subsequent dual-domain propagation.The network uses deformable alignment technology to propagate optical flow information of neighboring frames bidirectionally, thus completing the optical flow field.
• High-efficiency design:ProPainter is designed with efficiency in mind, using GPU acceleration to achieve efficient image propagation and feature propagation.By simplifying and optimizing operations, the complex and time-consuming processing in the CPU center is reduced.

ProPainter's project address

• GitHub repository:https://github.com/sczhou/ProPainter
• Project address:https://shangchenzhou.com/projects/ProPainter/
• arXiv technical paper:https://arxiv.org/pdf/2309.03897

How to use ProPainter

• Environmental preparation:Ensure that Python and necessary libraries, such as PyTorch, are installed in your environment.Create and activate a virtual environment to isolate project dependencies.
• Dependency installation:Install the required Python libraries according to the requirements of the ProPainter project.
• Code Acquisition:Clone ProPainter's GitHub repository to your local machine using Git.
• Weight Download:Download the pre-trained weight file. The weights are the parameters after the model has been trained.
• Data preparation:Prepare the video file to be repaired and the corresponding mask image (if there are specific areas that need to be repaired).
• Configuration file settings:Adjust the parameters in the configuration file according to the data and requirements.
• Model Inference:The trained model is used to perform inference on the video to achieve functions such as video repair, object removal, or video completion.

Application scenarios of ProPainter

• Film and video productionIn post-production video editing, ProPainter can remove unwanted objects or signs, such as billboards or continuity errors, while also filling in missing parts of the image due to technical issues.
• Historical video data restorationFor old or damaged video data, ProPainter can fill in missing frames or regions, restoring the original appearance of the video.
• Social media content creationContent creators can use ProPainter to remove watermarks from videos, enhancing their visual appeal and creative expression.
• Virtual Reality and Augmented RealityIn VR and AR applications, ProPainter can expand the field of view of a video, creating a more immersive visual experience.
• Video surveillance and securityIn the field of video surveillance, ProPainter can remove obstructions or fill in blind spots caused by obstructions, improving the usability of surveillance videos.
• Medical Imaging AnalysisIn the medical field, ProPainter can be used to fill in missing parts of medical images, helping doctors make more accurate diagnoses.