BigMac - Xiaohongshu's open-source multimodal large-model pipelined parallel training framework

What is BigMac?

BigMac is an open-source multimodal large-model pipelined parallel training framework developed by the Dots Infra team at Xiaohongshu. Addressing the challenge of balancing computational efficiency and memory usage in MLLM training, it proposes a secure nested pipeline paradigm. The framework uses a mature LLM pipeline as its backbone, embedding encoder and generator computations without disrupting the execution order. This achieves a 1.08–1.9x improvement in training speed while maintaining memory stability. It has been deployed in production as a core component of Dots multimodal training.

BigMac's main functions

• Dependency on safe nested pipelinesUsing the LLM pipeline as a stable backbone, encoder/generator computations are embedded at the locations where dependencies are satisfied to avoid cross-module bubble interference.
• Bounded mode activation of video memoryThe encoder and generator activate video memory down to O(1)The LLM activation behavior remains unchanged, supporting larger batch sizes without OOM.
• Decoupling of global scheduling and executionThe Scheduler generates a global operator table covering all ranks, microbatches, and modules; the Executor interprets the local sequence and distributes it to the corresponding backend.
• Seamless interface for production linesAlgorithm engineers only need to describe the module's input and output boundaries; BigMac automatically handles scheduling, handoff, and cross-device communication, keeping the model code modular.
• Schedule-aware toolchainBuilt-in profiler, simulator, and visualization tools; supports per-operator level trace export (compatible with Perfetto UI); quickly locates bubbles, slow operators, and communication waits.

The technical principles of BigMac

• Dependence on safe nested pipeline schedulingBigMac uses this as its base timeline. The scheduler analyzes the data dependencies between encoder forward, LLM forward, generator forward, and their corresponding backward operations, inserting modal computation only in "gap" periods where the input is ready and will not disrupt the LLM execution order. LLM continues according to its original schedule, and the activation of the encoder and generator can be released as soon as the gradients are ready. Backward operations are executed promptly, breaking the Pareto tradeoff that "computational efficiency inevitably leads to high GPU memory usage, and high GPU memory usage inevitably leads to more bubbles."
• Decoupling the global Operator table from the backendThe system explicitly encapsulates the scheduling strategy as a global operator table, containing forward/backward operators for all LLM/encoder/generators across all pipeline ranks, as well as module boundary communication. The execution layer only interprets local operator sequences; the LLM operator reuses the Megatron Core, and the modal operator retains its own data-parallel/FSDP implementation. Schedule adjustments do not require rewriting the model runtime.

Follow us on WeChat and reply with "open source",join inAI open source project discussion group

How to use BigMac

• Define module boundariesWrite the encoder_forward, llm_forward, and generator_forward functions in a modular manner, and declare the input and output of each module.
• Connect to BigMac interfaceBy registering modules through the PP-transparent interface, BigMac automatically handles pipeline stage division, activation handoff, gradient handoff, and cross-rank communication.
• Configure parallel strategySet parameters such as PP/VPP, number of microbatch, and module placement strategy.
• Run trainingThe Scheduler is started to generate a global plan, and the Executor executes the operator sequence on each rank.
• Performance DiagnosticsUse the built-in profiler to collect per-operator time and export the rank-aligned timeline to the Perfetto UI to analyze bubbles and bottlenecks; or use the simulator to quickly iterate parallel strategies before starting training.

BigMac's core advantages

• Breaking the Pareto FrontIt achieves both high computing efficiency and low video memory usage, eliminating the need to choose one over the other.
• LLM pipeline with zero interferencePreserve mature LLM schedules; encoder/generator latency fluctuations will not propagate along the pipeline.
• Video memory expandable: encoder / generator activates GPU memory O(1), supporting production-level large batch training.
• Algorithm friendlyThe model code does not need to be aware of the pipeline runtime, and experiments for single-card verification can be naturally extended to pipeline parallelism.
• Debuggable and deployableThe global schedule is visible and inspectable, and the accompanying trace/simulation toolchain reduces the cost of performance optimization.

BigMac's project address

• Project official website：https://dots-infra.github.io/BigMac/
• GitHub repository：https://github.com/Dots-Infra/BigMac/
• arXiv technical paper：https://arxiv.org/pdf/2605.25451

BigMac's Competitive Product Comparison

BigMac Application Scenarios

• Multimodal understanding model pre-trainingFor example, end-to-end pre-training of MLLMs for text and image comprehension and video comprehension requires efficient handling of pipelined collaboration between encoders such as ViT/Whisper and LLMs.
• Multimodal generative model trainingIt also includes understanding and generation paths (such as image generation and speech synthesis), and requires coordination of the bidirectional dependency between LLM and generators such as MMDiT/LDM.
• Large-scale batch trainingTo pursue a larger global batch size on memory-constrained clusters, and to avoid OOM caused by activation accumulation in traditional high-efficiency computing designs.
• Rapid Experimentation and IterationAlgorithm teams can validate model structures in single-card or DP environments and seamlessly extend them to pipeline parallelism via the BigMac interface without rewriting code.
• Training performance tuningThe engineering team used a schedule-aware profiler and simulator to locate pipeline bubbles and optimize parallel configuration and module placement strategies.