mHC - A new neural network architecture introduced by the DeepSeek team

What is mHC?

mHC (Manifold-Constrained Hyper-Connections) is a novel neural network architecture design method introduced by the DeepSeek team, addressing the stability issues of traditional Hyper-Connections (HC) architectures during large-scale training. mHC restores the identity mapping property of residual connections by projecting the residual connection space of HC onto a specific manifold, effectively avoiding gradient explosion or vanishing gradient problems. mHC utilizes the Sinkhorn-Knopp algorithm to project the residual connection matrix onto a manifold composed of two random matrices, ensuring that the mean of the signal remains constant during propagation while strictly normalizing the signal norm. mHC incorporates efficient infrastructure optimization techniques, such as kernel fusion, selective recomputation, and communication overlap, ensuring efficient implementation in large-scale models. Experiments show that mHC outperforms baseline models and HC in training stability, convergence speed, and downstream task performance, with only a slight increase in time overhead during large-scale training. As a general framework, mHC provides new ideas for deep learning architecture design and is expected to drive the evolution of next-generation infrastructure.

Main functions of mHC

• Restoring the identity mapping propertyBy projecting the residual connection space onto a specific manifold, mHC restores the identity mapping property of the residual connections, effectively solving the signal instability problem caused by the traditional hyperconnected architecture when expanding to multiple layers, and significantly improving the stability of large-scale training.
• Manifold constraints and signal specificationsBy utilizing a double random matrix manifold, mHC ensures that the signal maintains its mean during propagation and strictly regulates the signal norm, effectively avoiding the problems of gradient explosion or vanishing, and significantly enhancing the stability of signal propagation.
• Efficient infrastructure optimizationThrough techniques such as kernel fusion, selective recomputation, and communication overlap, mHC achieves efficient operation in large-scale models with minimal training overhead, ensuring high efficiency and scalability in practical applications.
• Improve model performanceExperiments show that mHC significantly outperforms baseline models and traditional hyperconnected architectures in multiple downstream tasks, especially in complex tasks such as BBH and DROP, improving the model's inference ability and overall performance.
• Scalability and flexibilitymHC, as a general framework, is applicable to various model scales and provides a new direction for exploring different manifold constraints in the future, which is expected to drive the evolution of the next generation of infrastructure.

mHC technical principles

• manifold projectionmHC constrains the properties of the connection matrix by projecting the residual connection matrix onto a specific manifold space (such as a Birkhoff polyhedron of birandom matrices). This projection ensures that the signal remains stable during propagation while preserving the ability to exchange information.
• Applications of double random matricesThe double random matrix has the property that the sum of its rows and columns is 1, which keeps the mean of the signal constant during propagation, strictly regulates the signal norm, and effectively avoids the problems of gradient explosion or vanishing.
• Sinkhorn-Knopp algorithmmHC utilizes the Sinkhorn-Knopp algorithm to perform manifold projection on the residual connection matrix. By iteratively normalizing the rows and columns, any non-negative matrix is adjusted into a doubly random matrix, achieving stable signal propagation.
• Recovery of identity mappingBy using manifold constraints, mHC restores the identity mapping property of residual connections, ensuring stable signal propagation in multilayer networks and solving the training instability caused by the lack of identity mapping in traditional hyperconnected architectures.
• High-efficiency computing and optimizationmHC combines kernel fusion, mixed-precision computing, and selective recomputation technologies to optimize computational efficiency and reduce memory access overhead, making the architecture highly efficient and scalable in large-scale model training.
• Stability of signal propagationmHC ensures signal stability during forward and backward propagation by constraining the residual connection matrix, significantly reducing extreme values of signal gain and improving the stability and convergence speed of model training.

mHC project address

• arXiv technical paperhttps://arxiv.org/pdf/2512.24880

Application scenarios of mHC

• Large-scale language model pre-trainingmHC can effectively improve the training stability of language models on large-scale datasets. It is suitable for pre-training tasks of ultra-large-scale language models such as 27B, and significantly improves the convergence speed and performance of the model.
• Multi-task learning and reasoningIn scenarios involving multiple downstream tasks (such as BBH, DROP, GSM8K, etc.), mHC enhances the model's inference ability and stability, helping the model achieve better results in complex tasks and improving the efficiency of multi-task learning.
• High-efficiency distributed trainingmHC, combined with optimized infrastructure design such as kernel fusion and communication overlap, is suitable for distributed training environments, and can significantly reduce training overhead and improve training efficiency, especially in large-scale clusters.
• Next-generation architecture evolutionmHC, as a general framework, provides a foundation for exploring novel neural network architectures, supports researchers in innovating under different manifold constraints and optimization strategies, and drives the evolution of next-generation infrastructure.
• Resource-constrained environmentBecause mHC introduces minimal computational overhead while maintaining performance, it is suitable for resource-constrained environments, such as mobile devices or edge computing scenarios, and can achieve efficient model deployment with limited computing resources.
• Academic research and theoretical explorationmHC provides a new perspective for studying signal propagation and architecture design in deep learning, and is suitable for in-depth exploration of neural network topology and optimization strategies in academic research.