ENEL - Shanghai AI Lab launches encoder-free 3D large-scale multimodal model

What is ENEL?

ENEL (Exploring the Potential of Encoder-free Architectures in 3D LMMs) is an innovative encoder-free 3D large-scale multimodal model (3D LMM) that addresses the limitations of traditional encoder architectures in 3D understanding tasks. ENEL eliminates the 3D encoder, directly converting point cloud data into discrete point markers, which are then concatenated with text markers and input into a large-scale language model (LLM). The model achieves efficient semantic encoding and geometric understanding through two key strategies: first, a semantic encoding strategy embedded in the LLM, extracting high-level semantics through hybrid semantic loss; and second, a hierarchical geometric aggregation strategy, enabling the LLM to focus on local details of the point cloud.

ENEL's 7B model excels on multiple 3D tasks, including 3D object classification, 3D object captioning generation, and 3D visual question answering (VQA). In the Objaverse benchmark, ENEL-7B achieves a GPT score of 50.92% for captioning generation, 55.0% for classification, and 42.7% for VQA on the 3D MM-Vet dataset, all comparable to existing 13B models (such as ShapeLLM). ENEL's encoderless architecture performs exceptionally well in semantic encoding, better capturing the semantic relevance between point clouds and text.

ENEL's main functions

• encoderless architectureENEL eliminates the traditional 3D encoder, directly converting point cloud data into discrete point tags via a tag embedding module. These tags are then concatenated with text tags before being input into the LLM. This avoids the point cloud resolution limitations and semantic embedding mismatch issues commonly found in encoder architectures.
• Advanced semantic extractionENEL uses an LLM embedding semantic encoding strategy to introduce hybrid semantic loss during the pre-training stage, which can extract high-level semantic features of point clouds while preserving key geometric structures.
• Local geometric perceptionDuring the instruction tuning phase, ENEL employs a hierarchical geometric aggregation strategy, enabling the LLM to proactively perceive local details of the point cloud. Through aggregation and propagation operations, local geometric information is integrated into the early layers of the LLM, achieving a refined understanding of complex 3D structures.
• Multi-task 3D understandingENEL performs exceptionally well on multiple 3D tasks, including 3D object classification, 3D object captioning generation, and 3D visual question answering (VQA). The 7B model achieved a classification accuracy of 55.0% and a captioning generation GPT score of 50.92% on the Objaverse benchmark, comparable to the performance of the 13B ShapeLLM.
• High-efficiency semantic alignmentENEL achieves efficient semantic alignment between point clouds and text modalities through an encoder-free architecture, which can better capture the semantic correlation between the two and provide a more powerful semantic foundation for 3D multimodal tasks.

ENEL's technical principles

• LLM-embedded Semantic EncodingDuring the pre-training phase, ENEL proposed a Hybrid Semantic Loss by exploring different self-supervised point cloud losses (such as mask modeling loss, reconstruction loss, contrast loss, and knowledge distillation loss). This loss function can embed high-level semantic information of the point cloud into the LLM, replacing the function of traditional 3D encoders.
• Hierarchical Geometry AggregationDuring the instruction tuning phase, ENEL introduced a hierarchical geometric aggregation strategy. This strategy incorporates inductive bias into the LLM by aggregating and propagating point clouds in early layers, allowing it to focus on local details of the point cloud. Specifically, it uses farthest point sampling (FPS) and k-nearest neighbor (k-NN) algorithms to downsample and aggregate the point cloud, gradually integrating fine-grained semantic information.

ENEL's project address

• Github repository:https://github.com/Ivan-Tang-3D/ENEL
• arXiv technical paper:https://arxiv.org/pdf/2502.09620

Application scenarios of ENEL

• 3D object classificationENEL can efficiently classify 3D objects and is suitable for fields such as industrial automation, robot vision and autonomous driving. It can quickly identify and classify complex 3D objects.
• 3D object caption generation:It can be used to generate descriptive text for 3D models, helping users quickly understand key information in 3D scenes, and is suitable for virtual reality (VR), augmented reality (AR), and 3D modeling tools.
• 3D Visual Question Answering (VQA)It can answer questions related to 3D scenes, such as in medical image analysis, architectural design, and education, helping users quickly obtain key information from 3D data.
• Understanding complex 3D geometrySuitable for scenarios requiring precise understanding of complex geometries, such as aerospace, automotive manufacturing, and jewelry design, it helps engineers and designers better understand and optimize 3D models.