Stanford University's CS336 course, "Language Modeling from Scratch,"

Stanford University's CS336 course, "Language Modeling from Scratch," is a course focused on language model development.Deep learningThe course provides students with a comprehensive understanding of the language model development process through hands-on practice, from data collection and model building to training and evaluation. Borrowing from operating systems courses, it requires students to build a complete language model from scratch, including implementing key components such as a tokenizer, Transformer architecture, and optimizer.

The course requires students to have strong Python programming skills.Deep learningAnd system optimization experience, as well as linear algebra, probability and statistics andMachine LearningThe course covers fundamental knowledge. Assignments are comprehensive, encompassing various aspects from basic model training to system optimization, data processing, and model alignment. It also provides recommendations for GPU computing resources to assist students.High efficiencyComplete practical tasks. Through this course, students will gain a deep understanding of the inner workings of language models and learn how to optimize them to meet the challenges of large-scale training and real-world applications.

• Course ContentStanford University's CS336 course aims to provide students with a comprehensive understanding of the language model development process, including data collection and cleaning, Transformer model building, model training, and evaluation. Inspired by operating systems courses, the course requires students to build a complete language model from scratch.
• Course ObjectivesThrough hands-on practice, students will gain a deep understanding of the various components of a language model and learn how to optimize the model to improve performance.High efficiencyRate and performance.

• Prerequisite knowledge:
  • Python proficiencyMost assignments require the use of Python, and students need to have strong Python programming skills.
  • Deep learningand system optimization experienceRequires familiarity with PyTorch and basic system concepts, such as memory hierarchy.
  • Mathematical FoundationsStudents need to master the basic knowledge of linear algebra, probability theory, and statistics.
  • Machine LearningBaseNeed to knowMachine LearningandDeep learningThe basic concept.
  • Homework 1Implement the basic components of the Transformer language model (word segmenter, model architecture, optimizer) and train a minimal language model.
  • Homework 2: Use advanced tools to perform performance analysis and optimization on the model, implement a Triton version of FlashAttention2, and build distributed training code.
  • Homework 3Understand each component of the Transformer and extend the patterns by fitting the model through the training API.
  • Homework 4The original Common Crawl data is transformed into data that can be used for pre-training, and then filtered and deduplicated to improve model performance.
  • Homework 5: Apply supervised fine-tuning and reinforcement learning to train language models to solve mathematical problems, and optionally implement safe alignment methods.

  Course website address

  • Official website address: https://stanford-cs336.github.io/spring2025/
  • GPU computing resourcesStudents can use the GPU resources of cloud service providers to complete assignments; the course provides several...recommendCloud service options.
  • Academic integrityStudents can useAIThe tool is for consulting on low-level programming problems or advanced conceptual problems, but direct use is prohibited.AITools solve problems.
  • Assignment SubmissionAll assignments are submitted through Gradescope, and late submissions are allowed, up to a maximum of 3 days.

  Kimi's "New PPT Assistant," a brand new self-developed tool by Kimi.freeAIPPT Generator

  Huawei launches HarmonyOSintelligentbodyFramework White Paper (PDF file)