AI Engineering Hub (PDF file) - AI Tutorial Materials

AI Engineering Hub isopen sourceThe project provides information on large language models (LLM), Search Enhancement Generation (RAG) and AI intelligentbodyIn-depth tutorials and practical examples for application setup.AI Engineering Hub includes theoretical explanations and provides numerous actionable code examples to help users.fastGetting Started. The project has garnered 12.6K stars on GitHub, and the core tutorial has been compiled into a 500+ page PDF for easy learning.AI Engineering Hub is suitable for beginners, practitioners, and researchers, and encourages community contributions to drive its development. AI Technological development.

Get theAI The original PDF file from Engineering Hub is available by scanning the QR code and replying. 20250713

AI Engineering Hub Course Content Overview

AI Engineering Hub course materials are comprehensive.practicalData Science andMachine LearningThis guide covers core topics from basic to advanced levels. The full text is divided into two main sections:

• Deep learningThis includes cutting-edge paradigms such as transfer learning, federated learning, and multi-task learning, as well as optimization techniques such as mixed-precision training and gradient checkpointing, and delves into large language models (…).LLMFine-tuning and deployment strategies for s).
• classicMachine LearningIt covers traditional methods such as feature engineering, regression analysis, decision trees, and clustering, combines statistical theory with practical cases (such as handling data drift and missing values), and reveals common pitfalls (such as the trap of random partitioning).

This resource provides data analysis tools (such as Pandas and SQL), visualization techniques (such as Sankey diagrams and QQ charts), and advanced uses of Python object-oriented programming. The entire text combines code examples, visualizations, and experimental comparisons, balancing theory and practice, making it suitable for learners of all levels. The accompanying assessment tool helps readers locate the most relevant chapters.High efficiencyImprove skills.

AI Engineering Hub Course Content

Deep LearningDeep learning)

Learning Paradigm

• Transfer learning, fine-tuning, multi-task learning, and federated learning:
  • Transfer learning: Based on the pre-trained model (base model), after training on the relevant task, the last few layers are replaced and the weights of other layers are frozen to adapt to new tasks with small data.
  • Fine-tuningAdjust some or all of the weights of the pre-trained model to directly adapt to new data.
  • Multi-task learning (MTL)The shared layer handles multiple tasks, while task-specific branches are trained independently, improving generalization and saving computational resources.
  • Federal LearningTraining models on distributed devices aggregates parameters rather than data, thus protecting privacy (such as mobile user data).
  • PyTorch implementation: shared layer + task branching, gradient accumulation and update, dynamic task weight allocation (e.g., adjusted according to validation accuracy).

  Runtime and memory optimization

  • momentumGradient moving average is used to reduce oscillations and accelerate convergence (hyperparameters need to be tuned).
  • Mixed precision trainingHybrid use of float16 and float32: Use float16 to accelerate forward/gradient calculations, and use float32 to maintain accuracy for weight updates. Loss scaling is required to avoid gradient vanishing.
  • Gradient checkpointsActivation values are stored in segments and recalculated during backpropagation, saving 50-60% of memory (at the cost of 15-25% of time).
  • gradient accumulation:
    • Accumulate gradients in multiple batches during mini-batch training and then update them to simulate the effect of large-batch training (without reducing time, but reducing memory pressure).
    • 4 Strategies for Multi-GPU Training
      • Model parallelLayers are split across different GPUs, requiring frequent communication.
      • Tensor Parallel: Decomposing single tensor operations (such as matrix multiplication in blocks).
      • Data ParallelismModel replication, gradient aggregation after data sharding training.
      • parallel pipelineMicro-batch pipeline processing improves GPU utilization.

      Miscellaneous

      • Label smoothingAdjust the label distribution, reduce the probability of the true class, distribute the margin evenly, and prevent the model from being overconfident (improving generalization but reducing prediction confidence).
      • Focus lossTo address class imbalance, the loss contribution of easily classified samples is reduced by using a weighting factor (γ) and the inverse class frequency (α).
      • How Dropout Actually Works? (Dropout principle)During training, neurons are randomly discarded, and the retained activation values are scaled by 1/(1-p) to maintain inference consistency (p is the discard rate).
      • The problem of Dropout in CNNThis disrupts the spatial correlation of convolutional layers and is recommended for use in fully connected layers.
      • The true role of hidden layers and activation functionsExplain how the hidden layer extracts feature levels and how the activation function introduces non-linearity.
      • Shuffle data before trainingTo avoid the data order between batches affecting model convergence.

      Model compression

      • Knowledge distillationTeachers use models to guide students' models, compressing model size.
      • Activate pruningPruning low-importance neurons reduces computational cost, but importance needs to be dynamically assessed.

      deploy

      • Deploying models from Jupyter Notebook: Simplify the model migration process from development to production.
      • Production Environment Model Testing MethodsThis includes production environment verification methods such as A/B testing and shadow deployment.
      • Version control and model registrationTrack model iterations and manage production environment releases.

      Large Language Model

      • Where did the GPU Memory Go? (GPU Memory Management)Analysis and TrainingLLMThe bottleneck of video memory usage and optimization strategies.
      • Full model fine-tuning vs. LoRA vs. RAGComparison of applicable scenarios: Full parameter fine-tuning consumes a lot of resources, LoRA (low-rank adaptation)High efficiencyRAG (Retrieval Enhancement) dynamically expands knowledge.
      • 5 typesLLMFine-tuning technologyThis includes a detailed explanation of lightweight fine-tuning techniques such as adapter fine-tuning and prefix fine-tuning.

      Classical MLMachine Learning)

      ML Basics

      • Time complexity of 10 algorithmsCompare the training/inference time complexity of 10 algorithms, including SVM and Random Forest.
      • 25 Key Mathematical DefinitionsIt covers core concepts such as probability and linear algebra.
      • How to reliably improve multi-class probability modelsThe multi-class probability output is calibrated using methods such as Platt Scaling.
      • Reasons why model improvements may be ineffective:Pitfalls such as data quality and selection of evaluation indicators.
      • Loss functions of 16 algorithms: Loss function formulas, including logistic regression, SVM, etc.
      • 10 common loss functionsFor example, the applicable scenarios for cross-entropy and Huber loss.
      • How to correctly use training/validation/test setsData partitioning strategies and preventing information leakage.
      • 5 cross-validation techniquesImplementation and selection of k-fold, leave-one-out method, etc.
      • Post-cross-validation stepsModel selection and final evaluation process.
      • Double decrease and bias-variance tradeoffExplain the nonlinear relationship between model complexity and generalization.

      Statistical Foundations

      • The difference between MLE and EMMLE is suitable for complete data, while EM handles latent variables (such as clustering).
      • Confidence interval and prediction intervalExplain the differences between the two and their calculation methods.
      • Why is OLS an unbiased estimate?Mathematical derivation and Gauss-Markov theorem.
      • Bartholomew's Distance: An indicator for measuring the similarity of probability distributions.
      • Why choose Mahalanobis distance?Mahalanobis distance offers advantages in considering the covariance structure.
      • 11 methods for testing the normality of dataThis includes Shapiro-Wilk, QQ images, etc.
      • Probability and LikelihoodThe connection between conceptual clarification and maximum likelihood estimation.
      • 11 key probability distributionsApplications of distributions such as Poisson and exponential.
      • Common Misconceptions about Continuous Probability DistributionsExplain the non-probabilistic meaning of the probability density function.

      Feature definition, engineering and selection

      • 11 types of variables in the datasetIt covers nominal variables (unordered categories), ordinal variables (ordered ranks), and continuous variables, guiding the selection of coding and analysis methods during data preprocessing.
      • Periodic feature encodingUse sin/cos encoding for periodic features (such as hours and months) in time series to preserve the continuity of the cycle (such as the proximity of 23:59 and 00:01).
      • Feature DiscretizationBinning continuous features (such as age segmentation) balances information loss and model robustness, and is suitable for linear models such as logistic regression.
      • 7 Classification Data Encoding MethodsThis includes one-hot encoding (sparse categories), target encoding (high cardinality categories), and hash encoding (memory optimization), which solve the problem of numericalizing category variables.
      • Disrupt feature importance assessmentBy randomly shuffling feature values, we can observe changes in model performance, quantify feature importance, and avoid overfitting to irrelevant features.
      • Probe Feature Selection MethodGenerate random noise features, filter out features with less importance than noise, and improve the model's generalization ability.

      Regression analysis

      • Mathematical Principles of MSEThe convexity of mean squared error (MSE) ensures that gradient descent converges to the global optimum and is sensitive to outliers (squared error).
      • Sklearn linear regression with no hyperparametersBecause it uses closed-form solutions (analytical solutions) to directly calculate weights, it does not require iterative optimization, but it cannot handle large-scale data (it requires gradient descent).
      • Poisson Regression vs. Linear RegressionPoisson regression processes count data (such as the number of events) using a log-connect function, while linear regression fits continuous target variables.
      • Dummy variable trapWhen encoding categorical variables, one category (such as "male" in gender) should be ignored to avoid multicollinearity caused by missing rank in the design matrix.
      • Generalized Linear Models (GLMs)Extend linear regression to exponential family distributions (such as binomial distribution and Poisson distribution), and link linear prediction with the target variable through a connection function.
      • Zero-inflation regressionFor count data with too many zero values (such as insurance claims), combine logistic regression (whether it is zero) and Poisson regression (count part).

      Decision Trees and Ensemble Methods

      • Random forest compressed into single treesBy extracting important rule paths, the forest is simplified into a single decision tree, sacrificing some accuracy to improve interpretability.
      • Decision trees inevitably overfitDue to its low bias (perfectly fitting the training data), variance needs to be controlled through pruning, limiting depth, or ensemble methods (such as random forests).
      • AdaBoost PrincipleIteratively train weak classifiers (such as shallow trees), adjust sample weights to focus on misclassified samples, and use weighted voting to improve overall performance.
      • Out-of-bag (OOB) verificationUsing the unsampled data (approximately 37%) in the random forest as the validation set, the model can be evaluated without additional data splitting.

      Dimensional reduction

      • PCA Variance InterpretationWhen selecting principal components, 95% of the cumulative variance is retained, balancing the dimensionality reduction effect with information loss, and it is suitable for scenarios with high feature correlation.
      • t-SNE vs. PCAt-SNE preserves local similarity through the t-distribution (suitable for visual clustering), while PCA preserves global variance (suitable for preprocessing).
      • Nuclear PCAIt maps data to a high-dimensional space through kernel functions (such as RBF) to handle nonlinear structures, and its computational complexity is higher than that of linear PCA.

      Section 2.7: Clustering

      • KMeans vs. Gaussian Mixture Model (GMM)KMeans hard assigns samples to the nearest cluster, while GMM soft assigns (probabilistic assignment). The latter is more flexible but requires more computational resources.
      • DBSCAN++ optimizationIt reduces the number of neighborhood queries and accelerates density clustering by sampling core points, making it suitable for large-scale data.
      • Hierarchical DBSCAN (HDBSCAN):automaticIt can determine the number of clusters and identify variable-density clusters, which is superior to the fixed parameter limitations of traditional DBSCAN.

      Correlation analysis

      • Limitations of Pearson correlation: Only detects linear relationships, Spearman rank correlation can capture monotonic nonlinear relationships (such as exponential relationships).
      • Anscombe Quartet InspirationThe same statistics (mean, variance, correlation coefficient) may correspond to completely different data distributions, which need to be verified by visualization.

      Missing data processing

      • Missing value types (MCAR/MAR/MNAR)MCAR (completely random missing) can be deleted, while MNAR (non-random missing) requires modeling the missing mechanism (such as survival analysis).
      • MissForest interpolationIt predicts missing values based on random forest iteration, and is suitable for mixed data types (numerical + categorical), outperforming mean/median imputation.

      Data analysis tools

      • Pandas/Polars/SQL/PySpark Syntax ComparisonFor example, grouping and aggregation in Pandas.groupby vs. PySparkgroupByIt helps migrate code across platforms.
      • GPU-accelerated Pandas (RAPIDS cuDF): Utilize GPU parallelization for data operations (such as sorting and joining) to accelerate large-scale data processing, while remaining compatible with the Pandas API.

      Advanced SQL Operations

      • Semi-join: Returns only rows from the left table that match the right table (e.g.,WHERE EXISTSWhen filtering fields that do not require the right table, it is more efficient than INNER JOIN.
      • Use NOT IN with cautionThe result may be empty if the right table contains NULL values; use [another method] instead.NOT EXISTSorLEFT JOIN + IS NULLSafer.

      Python object-oriented

      • Descriptors:pass__get__/__set__Methods control access to properties, enabling lazy evaluation and type validation (such as...).@property(Underlying mechanism).
      • PyTorch does not directly call forward():model()automaticTrigger forward, integrate hooks (such as pre/post processing) andautomaticDifferentiation preserves the integrity of the computational graph.

      AI Engineering Hub project address

      • GitHub repositoryhttps://github.com/patchy631/ai-engineering-hub

      AI Who is Engineering Hub?

      • beginnerFor the purpose of AI This course provides basic concept explanations and practical guides for interested but inexperienced learners.fastgetting Started.
      • Developers and practitionersProvides concrete code examples and practical cases for engineers and developers to help them apply their knowledge in real-world projects. AI technology.
      • ResearchersIt provides academic researchers with cutting-edge technology updates and research cases, facilitating the sharing and exchange of research findings.
      • Data Scientist: Providing a combination for data professionals AI A guide to data processing and model optimization in technology to improve work efficiency.
      • Technology enthusiastsProvided for technology enthusiastsup to date AI Practical examples of tools and frameworks to meet the needs of exploring new technologies.

      Get theAI The original PDF file from Engineering Hub is available by scanning the QR code and replying. 20250713

      Hengping Kimi K2, DeepSeek, Grok 4,Claude 4 typesLarge ModelWho is the true king?

      OpenAIIn Enterprisesartificialintelligent(PDF file) - AITutorial materials