Uni-AdaFocus - Tsinghua University Launches General-Purpose and Efficient Video Understanding Framework

What is Uni-AdaFocus?

Uni-AdaFocus is a general-purpose, high-efficiency video understanding framework developed by a research team from the Department of Automation at Tsinghua University. The framework uses an adaptive focusing mechanism to dynamically adjust the allocation of computing resources, enabling efficient processing of video content. Specifically, Uni-AdaFocus can intelligently filter video frames based on their importance, prioritizing frames containing key information and simplifying or skipping other frames, significantly reducing unnecessary computational overhead.

Main functions of Uni-AdaFocus

• Reduce time redundancyIt can dynamically locate and focus on key video frames related to the task, concentrate computing resources on these key frames, avoid processing all frames equally, reduce redundant calculations in the time dimension, and improve processing efficiency.
• Reduce spatial redundancyIn each video frame, only a portion of the spatial region is relevant to the task. Uni-AdaFocus can dynamically locate and focus on the task-relevant spatial regions within the video frame, focusing only on these regions to reduce spatial redundancy and further improve efficiency.
• Reduce sample redundancyThis approach allocates more computing resources to more challenging samples, differentiates the allocation of computing resources among different samples, and reduces computing investment for relatively "easy" videos, thereby achieving redundant modeling of the sample dimension and improving the overall processing effect.
• High-efficiency end-to-end trainingIt uses some mathematical methods to handle the problem of non-differentiable spatiotemporal dynamic computation, which can be conveniently and efficiently trained end-to-end without the need for more complex methods such as reinforcement learning.
• High compatibilityIt is compatible with a variety of readily available high-efficiency backbone networks, such as TSM and X3D, and can significantly improve the inference efficiency of these backbone networks.
• Inference costs can be flexibly adjustedUni-AdaFocus's inference cost can be adjusted online without additional training. It can fully utilize unstable computing resources or flexibly achieve the desired performance level with minimal power consumption by modifying the standard for calculation based on sample conditions.

The technical principle of Uni-AdaFocus

• Global EncoderLightweight feature extraction networks (such as MobileNet-V2) are used to coarsely process uniformly sampled video frames to obtain the overall spatiotemporal distribution information of the video, i.e., global features.
• Policy NetworkBased on global features extracted by the global encoder, keyframes and key regions within them are adaptively sampled to obtain patches of interest. The shape and size of the patch are adaptively determined according to the specific characteristics of the video frame.
• Local encoderA large-capacity neural network with a large number of parameters and high accuracy processes only the patches selected by the policy network, i.e., local features.
• ClassifierThe system aggregates global and local features frame by frame to obtain the optimal video understanding results, while using an early termination mechanism to model the redundancy of sample dimension calculations.

Uni-AdaFocus project address

• GitHub repository:https://github.com/LeapLabTHU/Uni-AdaFocus
• arXiv technical paper:https://arxiv.org/pdf/2412.11228

Application scenarios of Uni-AdaFocus

• Video recommendation systemUni-AdaFocus can provide users with more personalized recommendation services by efficiently identifying key information in video content.
• Video surveillance and security early warningBy monitoring abnormal behavior and emergencies in video streams in real time, Uni-AdaFocus can respond quickly and issue alerts to help relevant departments take timely measures.
• Intelligent editing and creationFor video creators, it can automatically identify key scenes and exciting moments in videos, helping creators to quickly complete editing and post-production.
• Education and TrainingUni-AdaFocus can accurately capture teachers' teaching actions and students' learning responses, improving teaching quality and learning outcomes.
• Health and Medical CareWhen using brain magnetic resonance imaging (MRI) to diagnose Alzheimer's disease and Parkinson's syndrome, Uni-AdaFocus can efficiently identify and analyze key features in MRI images, assisting doctors in making more accurate diagnoses.