ForgeTrain - A large-scale model pre-training framework jointly developed by Wallfacer Intelligence and Tsinghua University.

What is ForgeTrain?

ForgeTrain is the world's first production-grade large-scale model pre-training framework, jointly released by Facewall Intelligence, Tsinghua University, and the OpenBMB open-source community. It is entirely written by AI with zero human code intervention. The framework adopts the Forge Engineering three-stage methodology (standard setting → bit-by-bit alignment → performance overtaking), achieving a training speed 10% faster than Megatron on NVIDIA H100. It has successfully completed the pre-training process on Huawei Ascend chips, successfully training the MiniCPM5-1B model.

Main functions of ForgeTrain

• Production-grade large model pre-trainingIt supports distributed collaborative training using hundreds to thousands of GPUs and can be directly deployed for production use.
• Cross-hardware platform adaptationIt supports both NVIDIA H100 and Huawei Ascend series chips, and has completed the Ascend pre-training process.
• Performance surpassing human capabilitiesTraining speed on H100 is 10% faster than NVIDIA Megatron, and it saves 10% on computing power costs for the same task.
• Bit-by-bit consistency verificationThe AI-generated framework can produce numerical results that are completely consistent with the reference implementation (Megatron) under the same input, ensuring correctness.
• Automated assessment HarnessIt has a built-in automated testing and performance evaluation system, which transforms the criteria for running correctly and running fast into standards that can be automatically judged by the machine.

ForgeTrain's technical principles

• Forge Engineering Three-Stage Methodology:
  • Setting standards (Harnessing)Collect key operational data from Megatron to build an automated Harness benchmark and define correctness and performance baselines.
  • Bit-for-Bit ReplicationThe AI constructs a bit-by-bit consistent version with Megatron under Harness constraints, validating the AI's ability to accurately reproduce complex systems.
  • Constraint-free overpassingRemove binary consistency restrictions and switch to performance-oriented Harness, allowing AI to iterate autonomously in a larger search space, ultimately achieving speed breakthroughs.
• Code is a deeply customized productUnlike traditional software that treats code as a "asset" to be maintained in the long term, Forge Engineering decouples code into a deeply customized product that can be forged on demand—the same Harness can be forged into completely different, custom-optimized implementations on different hardware.
• Zero abstract lossThe versatility is retained in the Harness specification, and the high performance is kept in every forging, eliminating the traditional trade-off between versatility and customization.

How to use ForgeTrain

• Open source acquisitionThe code is fully open source; GitHub address: https://github.com/OpenBMB/ForgeTrain
• ContentsForgeTrain framework code (including H100 and Ascend versions) + Agent Harness complete toolchain (evaluation criteria, test specifications, agent orchestration scheme).
• Reproduction pathAny team can use Agent Harness to reproduce the entire process from bit-by-bit alignment to performance overtaking.
• Actual verificationWallfacer has completed the full pre-training of MiniCPM5-1B on Huawei Ascend chip using ForgeTrain, proving the feasibility of closed-loop operation.

ForgeTrain's core advantages

• AI in Manufacturing: The First Production-Level ValidationUnlike NVIDIA VibeTensor, which is explicitly labeled "non-production", Anthropic C compiler, and OpenAI Harness, ForgeTrain is the only AI generation framework that simultaneously achieves production-grade usability, superior performance, and complete open source.
• R&D efficiency increased by 100 times.The development of large model training frameworks will shift from being human-led to being AI-driven.
• Breaking the bottleneck of the domestic computing power ecosystemThere's no need to spend ten years building another CUDA; AI can instantly forge a dedicated software stack for each chip, enabling "software to keep up as fast as the chip is released."
• Cost savingsIt directly saves 10% of computing power costs for the same task.
• Level 4 AI Manufacturing AIIn the five-level evolutionary path of "AI creating AI", it reaches L4, that is, AI transforms its own infrastructure and recursion begins to compound.

ForgeTrain project address

• GitHub repositoryhttps://github.com/OpenBMB/ForgeTrain

Comparison of ForgeTrain and similar products

Application scenarios of ForgeTrain

• Large model pre-trainingIt directly replaces human-written training frameworks such as Megatron for training large production-grade models.
• Domestic computing power adaptationIt enables the rapid development of a dedicated high-performance training framework for domestically produced chips such as Huawei Ascend, breaking the monopoly of the CUDA ecosystem.
• AI research acceleratesReduce the R&D costs of large-scale models, and shift the annual capacity growth of large-scale models from a "human resource scale function" to a "computing power scale function".
• Software Engineering Paradigm ShiftAs the first example of Forge Engineering, it provides a methodological reference for the automatic generation of AI in other complex systems (compilers, databases, operating systems, etc.).
• End-side model trainingWallfacer has used it to train MiniCPM5-1B, which ranks first globally in the AA leaderboard for models smaller than 2B, and is suitable for the development of efficient edge-side models.