CAD-MLLM - A computer-aided design CAD model generation system jointly developed by ShanghaiTech University and multiple institutions.

What is CAD-MLLM?

CAD-MLLM, a computer-aided design (CAD) model generation system jointly developed by ShanghaiTech University, Transcengram, DeepSeek AI, and the University of Hong Kong, generates parametric CAD models based on various user inputs, such as text descriptions, images, point clouds, or combinations thereof. The system aligns and processes multimodal data using command sequences and large language models (LLMs) to construct complete CAD models. CAD-MLLM introduces a large-scale multimodal dataset called Omni-CAD and new evaluation metrics to comprehensively assess the topological quality and surface closure of the generated models. CAD-MLLM outperforms existing methods in performance and demonstrates high robustness to data defects.

Main functions of CAD-MLLM

• Multimodal input processingIt can process various input formats, including text descriptions, images, and point clouds, and generate CAD models based on the input.
• Parametric CAD Model GenerationThe system can generate parametric CAD models, which users can edit and adjust.
• Dataset Construction and Annotation: Introduce a dataset called Omni-CAD, which contains text descriptions, multi-view images, point clouds, and corresponding CAD command sequences.
• Evaluation Indicator InnovationThe system introduces new evaluation metrics to assess the topological quality and surface closure of the generated CAD model.
• robustnessIt exhibits high robustness when dealing with noise and missing data.
• Interactive DesignUsers can easily design CAD models based on simple instructions and illustrations, enabling even non-experts to realize their design ideas.

Technical principles of CAD-MLLM

• Command sequence representationThe command sequence of the CAD model is vectorized to form a data stream suitable for learning by large language models (LLMs).
• Multimodal data alignment: Use advanced LLMs to align vector representations of different modal data and CAD models, enabling the model to understand and process multiple inputs.
• Network architectureThe network architecture includes three modules: visual data alignment, point data alignment, and a large language model, and supports cross-modal input.
• Feature space sharingNon-text input is first processed based on a frozen encoder, and then the features are aligned in a shared large language model (LLM) feature space using a projection layer.
• Low-rank adaptation (LoRA) fine-tuningBased on integrated prompts and multimodal embedding, and by applying low-rank adaptation (LoRA) technology to fine-tune the LLM, an accurate CAD model is generated.
• Data augmentation methodsWe propose a data annotation process and data augmentation method to generate a new multimodal conditional CAD dataset, Omni-CAD.

CAD-MLLM project address

• Project official website:cad-mllm.github.io
• arXiv technical paper:https://arxiv.org/pdf/2411.04954

Application scenarios of CAD-MLLM

• Industrial design and manufacturingIt enables designers and engineers to quickly generate and modify complex industrial product CAD models, accelerating the product development process.
• Architecture and EngineeringArchitects and structural engineers can generate accurate CAD drawings from site photos or topographic data, improving the efficiency of design and planning.
• automotive industryAutomakers generate accurate CAD models of automotive parts from concept sketches or descriptions, optimizing design and manufacturing processes.
• AerospaceIn the aerospace field, CAD models of aircraft and spacecraft components and structures are generated from complex design requirements and performance parameters.
• Education and trainingFor students and beginners, this approach lowers the learning curve and improves teaching effectiveness.