UniPixel - A pixel-level multimodal large-scale model jointly launched by Hong Kong Polytechnic University and Tencent.

What is UniPixel?

UniPixel is the first unified pixel-level multimodal large-scale model developed by the Hong Kong Polytechnic University and Tencent ARC Lab team, focusing on fine-grained understanding and interaction of images/videos. It can complete three major tasks—object reference, pixel-level segmentation, and region reasoning—within a single model. Through an innovatively designed "object memory mechanism" and a unified visual encoding method, it achieves accurate tracking and semantic understanding of targets in videos. Based on the Qwen2.5-VL framework, the model supports three interaction methods: points, boxes, and masks. It surpasses traditional 72-parameter models in nine visual task benchmark tests. The code and online demo are open-source. Its core breakthrough lies in the deep integration of visual segmentation and language reasoning, solving the problem that traditional models cannot handle complex references and dynamic region understanding.

UniPixel's main functions

• Pixel-level visual language understandingIt focuses on pixel-level visual language understanding, enabling pixel-level alignment between visual signals and language semantics, and supports a variety of fine-grained tasks, including image/video segmentation, region understanding, and PixelQA tasks.
• Unified object reference and segmentationThe model seamlessly integrates object referencing and segmentation capabilities, generating relevant masks based on visual input prompts and performing subsequent inference based on these intermediate pointers during the inference process, achieving fine-grained pixel-level inference.
• Multitasking supportIt has demonstrated outstanding performance on multiple benchmark tests, including the ReVOS inference segmentation benchmark, MeViS, Ref-YouTube-VOS, RefCOCO/+/g datasets, and designed a new PixelQA task that requires the model to jointly perform object referencing, segmentation, and question answering.
• Flexible visual cue processingIt can flexibly handle visual cue input, generate masks and perform inference, support single-frame and multi-frame video region understanding, and mask-based question answering tasks.

UniPixel's technical principles

• Unified framework designUniPixel employs a unified framework that integrates object referencing and segmentation capabilities into a single model, enabling a leap from coarse-grained scene understanding to fine-grained pixel reasoning, thus providing a foundation for complex visual reasoning.
• Object memoryThe model includes an object memory that stores object features extracted from the referential task, providing contextual information for subsequent segmentation and inference tasks and enhancing the model's performance on pixel-level tasks.
• Multi-stage training strategyA multi-stage training strategy is adopted, which includes pre-training, fine-tuning for the algebraic task, and fine-tuning for the segmentation task, to gradually improve the model's performance on pixel-level tasks and adapt to different task requirements.
• End-to-end mask generationThe model can directly generate pixel-level masks based on language descriptions, achieving deep integration of language and vision, and supporting a variety of fine-grained tasks, such as image/video segmentation and region understanding.
• Flexible visual cue processingIt can flexibly handle visual cue input, generate masks and perform inference, support single-frame and multi-frame video region understanding, and mask-based question answering tasks, adapting to different scenario needs.
• Strong reasoning abilityIn the VideoRefer-Bench-Q question answering task, the UniPixel-7B model achieved an accuracy of 74.1%, surpassing several strong benchmark models, including GPT-4o, demonstrating its powerful capabilities in complex visual reasoning tasks.
• Model weights and dataset providedIt provides model weights for both UniPixel-3B and UniPixel-7B versions, as well as raw images/videos and preprocessed annotations for 23 reference/segmentation/QA datasets, offering rich resources for research and applications.
• Training and assessment supportThe codebase supports training and evaluation on 23 datasets and benchmarks, and features flexible hardware setups, efficient training techniques, custom base LLM and dialogue templates, as well as monitoring of the training process via Tensorboard/Wandb, making it convenient for users to use and optimize.

UniPixel's project address

• Project official websitehttps://polyu-chenlab.github.io/unipixel/
• Github repositoryhttps://github.com/PolyU-ChenLab/UniPixel
• HuggingFace datahttps://huggingface.co/datasets/PolyU-ChenLab/UniPixel-SFT-1M
• arXiv technical paper: https://arxiv.org/pdf/2509.18094
• Experience the demo onlinehttps://huggingface.co/spaces/PolyU-ChenLab/UniPixel

UniPixel application scenarios

• Image segmentationUniPixel can generate pixel-level masks of specific objects in an image based on language descriptions, making it suitable for scenarios requiring precise image segmentation, such as medical image analysis and target segmentation in autonomous driving.
• Video SegmentationIn the field of video processing, UniPixel can segment objects in videos in real time, supporting applications such as video editing, video surveillance, and augmented reality.
• Regional understandingIt can identify and segment specific regions in videos by understanding language descriptions, and can be used for video content analysis, intelligent monitoring systems, and background segmentation in video conferencing.
• Question and Answer TaskUniPixel supports PixelQA tasks, which can answer questions based on verbal descriptions and visual information, making it suitable for scenarios such as education, intelligent customer service, and information retrieval.
• Multimodal interactionIn scenarios that require interaction by combining visual and linguistic information, such as smart assistants, virtual reality, and game development, UniPixel can provide a more natural and accurate interactive experience.
• Intelligent monitoringIn the field of security monitoring, UniPixel can identify and segment specific objects or areas in surveillance videos in real time, improving the intelligence level of the monitoring system.