AB
AB
AiBoss

All Categories

project

NEO - A brand-new multimodal model architecture open-sourced by SenseTime and Nanyang Technological University.

NEO is a novel multimodal model architecture jointly developed by SenseTime and Nanyang Technological University. As the first native multimodal architecture (Native VLM), NEO breaks through the limitations of traditional multimodal models through deep innovation in its underlying architecture...

What is NEO?

NEO is a novel multimodal model architecture jointly developed by SenseTime and Nanyang Technological University. As the first native multimodal architecture (Native VLM), NEO breaks through the limitations of traditional multimodal models through deep innovation in its underlying architecture. Core innovations include native patch embedding, which captures image details more precisely; native 3D rotational position encoding (Native-RoPE), which perfectly adapts to the natural structure of images and text; and native multi-head attention, which enhances the model's ability to understand complex image-text relationships. NEO excels in data efficiency, performance, and inference cost-effectiveness, achieving top-tier visual perception capabilities with relatively little data, and has achieved excellent results in multiple authoritative evaluations. SenseTime has open-sourced both 2B and 9B versions of the NEO model, promoting the industrial application of native multimodal technology and building the next-generation multimodal technology standard.

NEO's main functions

  • Native multimodal fusionNEO achieves deep integration of images and text through its underlying architecture design, breaking through the modular limitations of traditional multimodal models and enabling more natural processing of mixed image and text content.
  • Efficient data utilizationWith only a small amount of data (such as 390 million image text examples), NEO achieves top-tier visual perception capabilities, significantly improving data utilization efficiency and reducing training costs.
  • Excellent performanceIn multiple authoritative evaluations, NEO has demonstrated excellent performance, providing high-quality output for tasks such as image understanding, text generation, and graph-text reasoning.
  • High reasoning cost-effectivenessEspecially at small to medium parameter scales (such as 0.6B-8B), NEO performs well in edge deployment and inference efficiency, making it suitable for a wide range of practical application scenarios.
  • Open source collaboration and extensionSenseTime has open-sourced two NEO models, 2B and 9B, to encourage developers and researchers to further develop and apply this architecture, and to promote the industrialization of multimodal technologies.

NEO's technical principles

  • Native Patch EmbeddingBy using bottom-up continuous mapping, image pixels are directly embedded into the model, avoiding the discretization process of traditional image tokenizers, capturing image details more precisely, and improving image modeling capabilities.
  • Native 3D Rotational Position Encoding (Native-RoPE)It innovatively decouples the three-dimensional spatiotemporal frequency allocation of images and text, assigning high-frequency codes to images and low-frequency codes to text, better adapting to the natural structures of the two modalities and supporting spatial structure association in complex scenes.
  • Native Multi-Head AttentionWithin a unified framework, it simultaneously achieves autoregressive attention for text and bidirectional attention for vision, enhancing the model's ability to understand text-image relationships and supporting complex text-image mixed understanding and reasoning tasks.
  • Underlying architecture innovationStarting from the underlying architecture, it achieves deep integration of multimodal modes, rather than simple modular splicing, fundamentally breaking through the performance bottleneck of traditional multimodal models and improving the overall efficiency of the model.
  • Efficient Training and ReasoningThrough optimized architecture design, NEO exhibits higher efficiency in training and inference, especially with small to medium parameter scales, achieving lower computational costs and higher inference speeds, making it suitable for a wide range of practical applications.

NEO's project address

  • Github repositoryhttps://github.com/EvolvingLMMs-Lab/NEO
  • arXiv technical paperhttps://arxiv.org/pdf/2510.14979

NEO application scenarios

  • Image and text generationNEO can generate high-quality images based on text descriptions, or generate accurate text descriptions based on image content, making it suitable for creative design, content creation, and other fields.
  • Intelligent search and recommendationBy understanding the deep semantics of images and text, NEO can provide users with more accurate search results and personalized recommendations, thus improving the user experience.
  • Multimodal question answeringNEO can handle question-and-answer tasks that combine images and text, providing more accurate answers by combining image and text information, making it suitable for scenarios such as education and customer service.
  • Intelligent driving and robot visionNEO's efficient image understanding capabilities can be used for scene perception and object recognition in intelligent driving, as well as environmental understanding and navigation in robot vision.
  • Industrial Inspection and MonitoringNEO can quickly and accurately identify anomalies and defects in images, making it suitable for quality inspection and monitoring systems in industrial production.
  • Medical image analysisNEO can assist doctors in analyzing and diagnosing medical images, and provide more comprehensive diagnostic suggestions by combining medical record text information.
Back