Zhejiang University's "Fundamentals of Large Models" textbook (PDF file)

《Large Model"Fundamentals" provides a comprehensive introduction to the large language model (LLMThis paper covers the fundamentals, architecture design, training optimization, and application practices of language models. Starting with the basic theories of language models, it delves into model architectures based on statistics, RNNs, and Transformers, focusing on the architectural types of large language models (such as Encoder-only, Encoder-Decoder, and Decoder-only) and their representative models (such as BERT, T5, etc.).GPT(Series). The book explains in detail.PromptEngineering, parametersHigh efficiencyThis course covers key technologies such as fine-tuning, model editing, and retrieval enhancement generation, demonstrating their application in various scenarios through real-world examples. Through systematic learning and practical guidance, it aims to help readers gain a deep understanding and effectively apply large language modeling techniques.

Get theLarge ModelBaseOriginal PDF report file. Scan the QR code to follow and reply: 20250915

Language Model Fundamentals

• Language Model Based on Statistical MethodsThis section introduces the n-gram model and its statistical principles (Markov hypothesis, maximum likelihood estimation).
• RNN-based language modelExplaining loopsNeural NetworksThe structure of RNNs, training problems (vanishing/exploding gradients), and their applications in language modeling.
• Transformer-based language model: A detailed analysis of the Transformer architecture (self-attention, FFN, layer normalization, residual connections) and its application in language models.
• Sampling methods for language modelsDecoding strategies include greedy search, beam search, Top-K sampling, Top-P sampling, and temperature mechanism.
• Evaluation of language modelsIntroduces internal assessments (such as perplexity) and external assessments (such as BLEU, ROUGE, BERTScore, G-EVAL).

Large Language Model Architecture

• Big Data + Large Model → NewintelligentThis paper analyzes the impact of model size and data size on model capabilities and introduces Scaling Laws (Kaplan-McCandlish, Chinchilla).
• Overview of the architecture of large language modelsCompare the attention mechanisms and applicable tasks of three mainstream architectures: Encoder-only, Encoder-Decoder, and Decoder-only.
• Encoder-only architectureUsing BERT as an example, this paper introduces its structure, pre-training tasks (MLM, NSP) and derivative models (RoBERTa, ALBERT, ELECTRA).
• Encoder-Decoder ArchitectureTaking T5 and BART as examples, this paper introduces their unified text generation framework and diverse pre-training tasks.
• Decoder-only architectureDetailed introduction GPT series(GPT-1 to GPT-4) and the development and characteristics of the LLaMA series (LLaMA1/2/3).
• Non-Transformer architectureThis section introduces state-space models (SSMs) such as RWKV and Mamba, as well as the test-time training (TTT) paradigm.

Prompt project

• Prompt Project Overview:definition Prompt and Prompt The project introduces the word segmentation and vectorization process (Tokenization, Embedding).
• In-Context Learning (ICL)This course introduces zero-shot, single-shot, and few-shot learning, as well as example selection strategies (similarity and diversity).
• Chain-of-Thought (CoT)Explain the three modes of CoT: Step-by-Step (CoT, Zero-Shot CoT, Auto-CoT), Think Before You Act (ToT, GoT), and Self-Consistency.
• Prompt SkillIntroduction includes specifications Prompt Write and summarize questions appropriately, use CoT when appropriate, and make good use of psychological suggestion (role-playing, scenario immersion).
• Related ApplicationsThe introduction covers based onLarge Modelofintelligentbody(Agent), data composition, Text-to-SQL,GPTS et al.

parameterHigh efficiencyFine-tuning

• parameterHigh efficiencyFine-tuning introductionThis section introduces two mainstream methods for downstream task adaptation—context learning and instruction fine-tuning—and leads to the discussion of parameters.High efficiencyPEFT (Precision Adjustment Technique) is explained in terms of cost reduction and improvement.High efficiencyAdvantages in terms of efficiency.
• Parameter appending method: A detailed explanation of how to achieve this by adding new, smaller trainable modules to the model structure.High efficiencyFine-tuning methods include adding to the input (such as...)PromptThe implementation and advantages of adding to the model (such as Prefix-tuning and Adapter-tuning) and adding to the output (such as Proxy-tuning).
• Parameter selection methodThis section introduces methods for fine-tuning only a subset of model parameters, categorized into rule-based methods (such as BitFit) and learning-based methods (such as Child-tuning). By selectively updating parameters, these methods reduce computational burden and improve model performance.
• Low-rank adaptation methodDetailed explanation of how to implement the original weight update matrix by approximating it with a low-rank matrix.High efficiencyThe fine-tuning methods are introduced, with a focus on LoRA and its variants (such as ReLoRA, AdaLoRA, and DoRA), and the parameter efficiency and task generalization ability of LoRA are discussed.
• Practice and ApplicationThis section introduces the usage and related techniques of the HF-PEFT framework, demonstrates application cases of PEFT technology in tabular data querying and analysis, and proves that PEFT can improve...Large ModelEffectiveness in terms of performance for specific tasks.

• Model Editing IntroductionThis paper introduces the concept, definition, and nature of model editing, and elaborates on its importance in correcting biases, toxicities, and knowledge errors in large language models.
• Classic methods of model editingModel editing methods are divided into external extension methods (such as knowledge caching and additional parameter methods) and internal modification methods (such as meta-learning and localization editing methods), and representative works of each type are introduced.
• Additional parameter method: T-PatcherThis section provides a detailed introduction to the T-Patcher method, which achieves precise control over the model output by attaching specific parameters to the model. It is suitable for applications requiring...fast, and scenarios for accurately correcting specific knowledge points in the model.
• Positional editing method: ROMEThis section provides a detailed introduction to the ROME method, which achieves precise control over the model's output by locating and modifying specific layers or neurons within the model. It is suitable for scenarios that require deep modification of the model's internal knowledge structure.
• Model editing applicationThis section introduces the practical applications of model editing in areas such as accurate model updates, protecting the right to be forgotten, and enhancing model security, showcasing the application potential of model editing technology in different scenarios.

Search Enhancement Generation

• Introduction to Search Enhancement GenerationThis section introduces the background and components of search enhancement generation, and elaborates on its application in...Natural Language ProcessingThe importance and application scenarios of improving model performance by combining retrieval and generation in the task.
• Search Enhanced Generative ArchitectureThis section introduces the RAG architecture classification, black-box enhancement architecture, and white-box enhancement architecture, and compares and analyzes the characteristics and applicable scenarios of different architectures.
• Knowledge RetrievalThis section introduces methods for knowledge base construction, query enhancement, search engine and search efficiency improvement, and how to improve search performance by rearranging search results.
• Generative enhancementThis section introduces methods for when to enhance, where to enhance, how to enhance multiple times, and how to reduce costs and increase efficiency, and discusses application strategies for generative enhancements in different tasks.
• Practice and ApplicationIntroduction to setting upSimpleThe steps of the RAG system are explained, and examples of RAG in typical applications are shown to help readers understand and apply search enhancement generation techniques.

Get theLarge ModelBaseOriginal PDF report file. Scan the QR code to follow and reply: 20250915

《AI Agent"Illustrated Guide to Panoramic Technology Report" (PDF file)

Anthropic releases "ForintelligentbodyOfficial tutorial for "Programming Tools"