nanochat - Karpathy's open-source, low-cost, self-built ChatGPT full-stack project

What is nanochat?

nanochat is an open-source project released by AI expert Andrej Karpathy. It trains small language models with extremely low cost and high efficiency to achieve conversational functionality similar to ChatGPT.With just about $100 (4 hours of training using 8 H100 GPUs), a small model can be trained to engage in basic conversations, create stories/poems, and answer simple questions. Increasing the budget to $1000 (approximately 41.6 hours of training) significantly improves model performance, enabling it to solve simple math/coding problems and participate in multiple-choice tests.The project encompasses the entire process from data preparation, pre-training, mid-training, supervised fine-tuning (SFT), reinforcement learning (RL) to inference deployment. Approximately 8,000 lines of code implement end-to-end training, with concise and readable code suitable for learning and practice.

Main functions of nanochat

• Word segmenter trainingThis program uses the Rust language to train a tokenizer, which is responsible for converting text into a sequence of symbol codes.
• Pre-trainingPre-train large language models with the Transformer architecture on the FineWeb dataset and evaluate model performance using the CORE metric.
• Mid-term trainingMid-term training was conducted on the SmolTalk user-assistant dialogue dataset, multiple choice dataset, and tool usage dataset to adapt the model to dialogue scenarios.
• Supervisory fine-tuning (SFT)Supervised fine-tuning was performed on the World Knowledge Multiple Choice Dataset (ARC-E/C, MMLU), the Mathematics Dataset (GSM8K), and the Code Dataset (HumanEval) to improve the model's performance on specific tasks.
• Reinforcement learning fine-tuning (RL)The model was fine-tuned using reinforcement learning on the GSM8K dataset using the "GRPO" algorithm to further optimize model performance.
• Inference DeploymentIt enables efficient model inference, supports key-value caching, a simplified pre-filling/decoding process, and tool usage (a Python interpreter in a lightweight sandbox environment), and allows interaction with the model via CLI or a ChatGPT-like WebUI.
• Transcript generationGenerate a single Markdown-formatted report card that summarizes the entire training and inference process and presents the results in a "gamified" format.

nanochat's technical principles

• Minimalist code architectureThe entire project has only about 8,000 lines of code, is implemented using a single codebase, has very few dependencies, a clear structure, and is easy to understand and modify.
• Rust language tokenizerThis code uses the Rust language to train a word segmenter, which is responsible for converting text into a sequence of symbol codes to improve word segmentation efficiency and performance.
• Transformer architecture:Build large language models based on the Transformer architecture and learn language patterns and knowledge through pre-training.
• Data-driven trainingPre-training on datasets such as FineWeb allows the model to learn language expressions and knowledge through a large amount of text data.
• Mid-term training adaptation: Perform mid-term training on dialogue datasets such as SmolTalk to adapt the model to dialogue scenarios and specific tasks.
• Reinforcement learning optimizationThe "GRPO" algorithm is used to fine-tune reinforcement learning on a specific dataset to further optimize model performance.
• High-efficiency inference engineImplement an inference engine with KV caching, supporting pre-filling and decoding processes to improve inference efficiency.
• WebUI InteractionIt provides a ChatGPT-like web interface, allowing users to interact with the trained model through the WebUI.

nanochat project address

• Github repositoryhttps://github.com/karpathy/nanochat

Application scenarios of nanochat

• Individuals and TeamsSuitable for individuals or teams with strong cybersecurity awareness, enabling them to quickly establish encrypted communication channels within an internal network.
• Developers and technology enthusiastsIt serves as a practical platform for learning and researching P2P networks, encryption technologies, and command-line application development.
• Temporary working groupFor example, an emergency response team can quickly establish a communication network without a central server.
• Education and ResearchNanochat provides researchers and learners with a low-cost, easy-to-understand, and easy-to-improve LLM development platform.