AB
AB
AiBoss

All Categories

project

LLaMA-Mesh - A complex 3D modeling project for natural language generation jointly launched by Tsinghua University and NVIDIA.

LLaMA-Mesh is a project jointly launched by Tsinghua University and NVIDIA. It combines 3D mesh generation with Large Language Models (LLMs) to enable the direct generation of 3D models from text prompts. The project uses OBJ file format to store 3D meshes...

What is LLaMA-Mesh?

LLaMA-Mesh is a project jointly launched by Tsinghua University and NVIDIA. It combines 3D mesh generation with Large Language Models (LLMs) to enable the direct generation of 3D models using text prompts. The project uses OBJ file format to convert the vertex coordinates and face definitions of 3D meshes into text, and optimizes the processing using vertex quantization technology, allowing the model to understand and generate 3D meshes. LLaMA-Mesh can generate high-quality 3D meshes while maintaining strong language understanding and generation capabilities, providing a more intuitive and efficient new method for 3D content creation.

Main functions of LLaMA-Mesh

  • 3D Mesh GenerationGenerate the corresponding 3D mesh model based on the text prompts.
  • Grid Understanding: To understand and interpret the structure and features of 3D meshes.
  • Text-grid interleaved outputGenerates interlaced output of text and 3D meshes in dialogues to enable interactive design.
  • Maintaining language skillsWhile extending to 3D mesh generation, it maintains the model's text understanding and generation capabilities.

The technical principle of LLaMA-Mesh

  • 3D representation:
    • OBJ file formatThe vertex coordinates and face definitions of a 3D mesh are represented as plain text using the OBJ file format, which can be processed by a language model.
    • Vertex quantizationQuantizing vertex coordinates into a fixed number of intervals reduces the number of labels, allowing the model to handle longer sequences while preserving geometric details.
  • pre-trained model:LLaMA3.1-8B-Instruct uses a pre-trained model that has been adjusted with instructions, and has the ability to understand text prompts and generate 3D meshes.
  • 3D task fine-tuning:
  • Supervised Fine-tuning (SFT) datasetWe constructed a dataset containing text-3D pairs and interleaved text-3D dialogues, and fine-tuned the model to master 3D mesh generation skills.
  • Rules and LLM EnhancementBy combining rule-based methods and language model-based augmentation techniques to construct training data, the model's 3D understanding and generation capabilities can be improved.
  • Unified Model:
  • Text and 3D mesh unificationGenerate text and 3D meshes within a unified model to achieve multimodal content generation.

LLaMA-Mesh project address

Application scenarios of LLaMA-Mesh

  • Creative DesignDesigners can quickly generate 3D models of furniture, decorations, artworks, etc., accelerating the transformation from concept to prototype.
  • Game developmentGame developers can quickly generate 3D models of weapons, items, characters, etc. in games, improving the efficiency and richness of game design.
  • Education and TrainingIn the field of education, it can assist teaching by generating 3D models of geometric shapes, biological structures, historical artifacts, etc., to enhance students' understanding and learning experience.
  • Architecture and EngineeringArchitects and engineers generate building models and engineering components for design verification and visualization.
  • Virtual Reality (VR) and Augmented Reality (AR)In VR and AR applications, it generates realistic 3D scenes and objects, providing an immersive user experience.
Back