MatAnyone - A portrait video cutout framework jointly developed by Nanyang Technological University and SenseTime.

What is MatAnyone?

MatAnyone, developed by Nanyang Technological University's S-Lab and SenseTime, is an advanced framework for matting portrait videos against complex backgrounds, focusing on target-specific video matting tasks. MatAnyone utilizes a consistent memory propagation module and region-adaptive memory fusion technology to ensure semantic stability and fine-grained boundary details in core regions of video sequences. MatAnyone introduces a novel training strategy, directly supervising the matting head with large-scale segmentation data, significantly improving the model's stability and generalization ability in real-world scenarios. MatAnyone comes equipped with the high-quality, diverse training dataset VM800 and the more challenging test dataset YoutubeMatte, providing a solid foundation for model training and evaluation.

MatAnyone's main functions

• Stable target trackingStable tracking of the target object throughout the video, maintaining the integrity of the target in complex or blurry backgrounds.
• Fine-grained edge detail extractionIt supports the generation of high-quality alpha masks, especially excelling in boundary areas (such as hair, edges, etc.), providing image-level detail accuracy.
• Adaptable to various video typesMatAnyone can handle different types of video, including movies, games, smartphone videos, etc., and adapts to various frame sizes and media formats.
• Enhanced interactivityUsers can specify a target segmentation mask in the first frame to guide the entire video cutout process, enabling more precise interactive video editing.

MatAnyone's technical principles

• Consistent memory propagation:
  • Memory FusionThe CMP module adaptively fuses information from the previous frame based on an estimate of the alpha value change between the current and previous frames. For regions of "large change" (usually located at the boundaries), it relies more on information from the current frame; for regions of "small change" (usually located in the core region), it retains the memory from the previous frame.
  • Region AdaptationBased on a lightweight boundary region prediction module, the probability of change for each query token is estimated, enabling region-adaptive memory fusion, which significantly improves the semantic stability of the core region and the detail accuracy of the boundary region.
• Core Area Supervision:
  • Direct supervision of segmented dataTo overcome the scarcity of real-world video matting data, MatAnyone directly supervises the matting head using large-scale real-world segmentation data. It ensures semantic stability and detail accuracy by using pixel-level loss (Lcore) in the core region and an improved DDC loss (Lboundary) in the boundary region.
  • Improved DDC lossBased on adjusting the calculation method of DDC loss, it is more suitable for video matting tasks, avoiding the jagged and stepped edges produced by traditional DDC loss in the boundary area.
• New dataset and training strategy:
  • High-quality training datasetA new training dataset, VM800, is introduced, which is twice the size of the existing dataset VideoMatte240K and has higher quality in the core and boundary regions, significantly improving the training effect of the model.
  • Multi-stage trainingA multi-stage training strategy is adopted. First, the model is initialized on video matting data, then core region supervision is performed based on segmentation data, and finally boundary details are further optimized using image matting data.
• Network architecture:
  • encoderThe ResNet-50 encoder is used to extract features and generate queries and keys.
  • Object TransformerBased on the object transformer module, pixel-level memory is grouped according to object semantics to reduce noise caused by low-level pixel matching.
  • decoderThe decoder generates a high-precision alpha mask based on multi-level upsampling and skip connections.
  • Value EncoderThe predicted alpha mask and image features are encoded into values for use in updating the memory library.

MatAnyone's project address

• Project official website:https://pq-yang.github.io/projects/MatAnyone/
• GitHub repository:https://github.com/pq-yang/MatAnyone
• arXiv technical paper:https://arxiv.org/pdf/2501.14677

MatAnyone application scenarios

• Film and television post-productionUsed for background replacement and special effects compositing, it precisely extracts actors from the original background and replaces them with virtual or special effects backgrounds, enhancing the visual effects and creative possibilities of the image.
• Video conferencing and live streamingIn video conferencing and live streaming, people can be separated from complex backgrounds in real time and replaced with virtual or blurred backgrounds to enhance privacy protection and visual effects.
• Advertising and MarketingIn advertising video production, products or people can be cut out from the shooting background and replaced with a more attractive background to enhance the visual impact and appeal of the advertisement.
• Game developmentUsed in video content production within games, such as character animations and cutscenes, to precisely extract characters from the shooting background and replace them with game scenes, enhancing the game's immersive experience.
• Virtual Reality and Augmented RealityIn VR and AR applications, users or objects are extracted from real-world scenes and integrated into virtual environments to enhance user experience and interaction.