Oryx - A multimodal large language model jointly developed by Tencent, Tsinghua University, and Nanyang Technological University

What is Oryx?

Oryx is a multimodal large-scale language model (MLLM) jointly developed by Tsinghua University, Tencent, and Nanyang Technological University. It utilizes two core innovations to process visual data: the pre-trained OryxViT model and a dynamic compression module. OryxViT encodes images of arbitrary resolution into visual representations suitable for LLM, while the dynamic compression module compresses visual tags by a factor of 1 to 16 as needed. This allows Oryx to flexibly handle visual inputs of varying resolutions and durations, from high-definition images to ultra-long videos. Oryx demonstrates superior performance in multiple vision-language benchmarks, particularly in spatial and temporal understanding.

Oryx's main functions

• Native resolution processingOryx can handle visual input of any resolution, preserving all the details of the image, making it suitable for tasks requiring high-precision visual information.
• Dynamic compressionDepending on the task requirements, Oryx can dynamically compress visual data by 1 to 16 times, processing large-scale data such as long videos and improving computational efficiency.
• Multimodal understandingIt understands and analyzes images, videos, and 3D data, providing rich spatial and temporal understanding capabilities, and is suitable for a variety of visual-language tasks.
• Contextual SearchEnhance the contextual understanding of video content and retrieve specific information from a broad context.
• Spatial perceptionOryx can accurately grasp the position and relationship of objects in 3D space, enhancing the understanding of three-dimensional space.

Oryx's technical principles

• OryxViT modelA pre-trained visual encoder that converts images of different resolutions into visual representations suitable for processing by large language models.
• Adaptive Position EmbeddingOryxViT uses adaptive position embedding layers, allowing the model to handle images of different sizes without needing to be adjusted to a fixed resolution.
• Variable-length self-attention mechanismIt allows models to process visual data of different sizes in parallel, improving processing efficiency and flexibility.
• Regional attention operationsIn the dynamic compression module, region attention is used to manipulate the interactive high-resolution and low-resolution feature maps to mitigate the impact of downsampling.
• Mixed data training: Train the model on a mixed dataset that includes images, videos and 3D data to improve its performance on multimodal tasks.

Oryx's project address

• Project official websiteoryx-mllm.github.io
• GitHub repository:https://github.com/Oryx-mllm/Oryx
• HuggingFace model library:https://huggingface.co/spaces/THUdyh/Oryx
• arXiv technical paper:https://arxiv.org/pdf/2409.12961

Oryx application scenarios

• Intelligent monitoringBased on Oryx's video understanding capabilities, it enables real-time monitoring and analysis of events and activities in surveillance videos.
• autonomous drivingIn autonomous driving systems, Oryx helps analyze and understand the environment around the vehicle, providing more accurate visual recognition.
• Human-computer interactionOryx can understand image and video content, making human-computer interaction more natural and efficient.
• Content moderationOn social media and online platforms, Oryx helps automatically identify and filter inappropriate content.
• Video editing and enhancementOryx can automatically edit videos, such as video summarization and highlight generation.
• Education and trainingIn the education sector, Oryx provides intelligent analysis of image and video content to support teaching and learning.