VITRON - Skywork AI, in collaboration with the National University of Singapore and Nanyang Technological University, has developed a pixel-level visual large-scale language model.
VITRON is a pixel-level visual large-scale language model (LLM) jointly developed by Skywork AI, the National University of Singapore, and Nanyang Technological University. It can comprehensively understand and process static images and dynamic videos, enabling image and video comprehension...
What is VITRON?
VITRON is a pixel-level visual large-scale language model (LLM) jointly developed by Skywork AI, the National University of Singapore, and Nanyang Technological University. It comprehensively understands and processes static images and dynamic videos, performing image and video understanding, generation, segmentation, and editing. VITRON combines a front-end visual encoder with a back-end visual expert system, supporting various tasks from visual understanding to visual generation. VITRON uses a hybrid approach to deliver information, combining discrete text instructions and continuous signal embedding to achieve precise function calls. It also features a cross-task collaboration module to enhance synergy between different visual tasks.
VITRON's main functions
- Visual understandingThis includes question answering (QA) for images and videos, referring expression, and visual reasoning.
- Visual generationText-to-Image and Text-to-Video.
- Visual segmentationThis includes image and video segmentation tasks, such as instance segmentation and panoptic segmentation.
- Visual editingEdit images and videos, such as adding, replacing, removing, and changing colors.
- Interactive user input: Handles interactive input from users, such as clicks, drawing boxes or polygons, and drawing.
VITRON's technical principles
- Encoder-LLM-Decoder ArchitectureIt is based on a common encoder-large language model (LLM)-decoder architecture, where the encoder is responsible for processing image and video input, the LLM is responsible for semantic understanding and decision-making, and the decoder is responsible for performing specific visual tasks.
- Front-end vision - language encodingCLIP ViT-L/14@336px was used as the image and video encoder to process each frame of video using average pooling to obtain the overall temporal representation features. A region pixel-aware visual extractor was used as a sketch encoder to process the user's interactive input.
- Core LLMUsing Vicuna (7B, version 1.5) as an LLM, it processes inputs from linguistic and visual modalities, performs semantic understanding and reasoning, and generates decisions.
- Backend vision expertIt integrates a series of single vision experts, such as GLIGEN for image generation and editing, SEEM for image and video segmentation, ZeroScope and I2VGen-XL for text-to-video and image-to-video tasks, and StableVideo for video editing.
- Mixed method instruction passingA novel hybrid approach is introduced, based on discrete text instructions and continuous signal feature embedding, to ensure that LLM decisions are accurately transmitted to back-end modules.
VITRON's project address
- Project official website:vitron-llm.github.io
- GitHub repository:https://github.com/SkyworkAI/Vitron
- arXiv technical paper:https://arxiv.org/pdf/2412.19806
VITRON application scenarios
- Image editing assistance: To repair and enhance photos, such as removing unwanted objects or enhancing image colors.
- Video content creationGenerate video content from script text, including scene construction and character animation.
- Online education platformThe educational platform automatically generates teaching videos and images based on the teaching syllabus to assist in teaching.
- E-commerce visual marketingIt automatically generates product display videos for e-commerce platforms to enhance product appeal.
- News Media Content ProductionIt helps news organizations quickly generate visual reports of news events, including images and videos.