MiniMind - an open-source AI model training tool that can train a 25.8M small model in 2 hours.

What is MiniMind?

MiniMind is an open-source, ultra-small language model project that helps individual developers train their own language models from scratch at extremely low cost. MiniMind is designed for lightweight use; the smallest version requires only 25.8M parameters and is only 1/7000th the size of GPT-3, making it suitable for fast training on ordinary personal GPUs. MiniMind provides complete training workflow code, including pre-training, supervised fine-tuning, LoRA fine-tuning, reinforcement learning, and model distillation. It supports multimodal capabilities (such as the visual language model MiniMind-V) and is compatible with mainstream frameworks such as...transformersandpeftMiniMind has open-sourced high-quality datasets and custom tokenizers, making it suitable for LLM beginners to get started quickly.

MiniMind's main functions

• Model training with extremely low barriers to entry:
  • low costThe GPU rental cost is only 3 RMB (based on NVIDIA 3090).
  • Quick TrainingTraining from scratch takes only 2 hours.
  • Lightweight designThe minimum model has only 25.8M parameters, making it suitable for running on ordinary devices.
• Fully open sourceProvides complete training code, covering pre-training, supervised fine-tuning (SFT), LoRA fine-tuning, direct preference optimization (DPO), and model distillation. All core algorithms are implemented natively in PyTorch, without relying on third-party wrappers, making them easy to learn and extend.
• Supports multiple training techniques:
  • Hybrid Expert (MoE) ArchitectureDynamically allocate computing resources to improve the learning efficiency of small models.
  • Direct Preference Optimization (DPO)): No complex reward model is needed; the model output is optimized based on human preferences.
  • Multimodal extensionSupports visual multimodal (MiniMind-V) to enable image dialogue and image-text generation.

MiniMind's technical principles

• Transformer architectureThis is a Decoder-Only architecture based on Transformer, similar to the GPT series. Pre-normalization and RMSnorm normalization methods are used to improve model performance. The Swiglu activation function replaces ReLU to improve training efficiency.
• Hybrid Expert (MoE) TechnologyIntroducing a hybrid expert module into the feedforward network (FFN) dynamically allocates computing resources to different "experts." Based on sharing and isolation techniques, this enhances the learning ability and efficiency of small models.
• Lightweight training process:
  • Pretraining: Use cleaned, high-quality text data for unsupervised learning to accumulate language knowledge.
  • Supervisory fine-tuning (SFT)Supervised fine-tuning of the model based on dialogue templates to adapt it to chat scenarios.
  • LoRA fine-tuningIt updates a small number of parameters based on low-rank decomposition, enabling rapid adaptation to specific domains or tasks.
  • Direct Preference Optimization (DPO): Optimize the output quality of the model based on labeled data with human preferences.
  • Model distillation: Imitate the output of the large model to further improve the performance of the small model.

MiniMind's project address

• Project official website:https://jingyaogong.github.io/minimind/
• GitHub repository:https://github.com/jingyaogong/minimind
• HuggingFace model library:https://huggingface.co/collections/jingyaogong/minimind

MiniMind application scenarios

• AI beginners and studentsFor students and beginners interested in AI, to complete their graduation project or participate in academic research.
• Individual developers and independent developersFor individual developers with limited resources, this tool enables rapid development and deployment of AI applications, suitable for experimental and innovative projects.
• Vertical professionalsProfessionals in fields such as medicine, law, and education can develop tools such as medical consultation assistants, legal consultation tools, or educational tutoring systems.
• Small teams and entrepreneursFor small teams or entrepreneurs with limited resources to develop a minimum viable product (MVP) or explore new business directions.
• enthusiasts and creatives: Enthusiasts who are interested in technology but do not have a strong background in it, develop intelligent chatbots, generate creative content, or explore multimodal applications.