Relax - Xiaohongshu's open-source large-scale model reinforcement learning training engine

What is Relax?

Relax is an open-source reinforcement learning training engine for large models designed for all-modal and agentic scenarios, developed by the Xiaohongshu AI platform team. Built on Megatron-LM and SGLang, it employs a service-oriented fault-tolerant architecture (Ray Serve) and an asynchronous data bus (TransferQueue) to completely decouple the roles of Rollout, Actor, and Critic. Relax supports unified training for text, images, audio, and video, and has demonstrated stable convergence of four modal RL on the Qwen3-Omni-30B. In 16×H800 fully asynchronous mode, it achieves a 76% speedup compared to Colocate and a 20% speedup compared to veRL. It features minute-level fault recovery, elastic scaling, and R3 MoE support, and has been applied to post-training on the Qwen3 series.

The main functions of Relax

• Full-modal RL trainingIt uniformly supports end-to-end reinforcement learning post-training for text, images, audio, and video, and has verified stable convergence of the Qwen3-Omni series.
• Asynchronous training architectureBased on the TransferQueue data bus, the Rollout, Actor, and Critic roles are fully asynchronous and parallel, eliminating GPU idle waiting.
• Service-oriented fault toleranceEach RL role is deployed as an independent Ray Serve, with fault isolation and minute-level automatic recovery capabilities, supporting uninterrupted training.
• Elastic expansion and contraction capacity: Dynamically add or remove Rollout inference engines via HTTP REST API, supporting both intra-cluster and cross-cluster federated inference resource scheduling.
• Distributed weight synchronizationThe DCS service implements NCCL/TCP dual-channel low-latency weighted broadcasting, eliminating the need to roll back disk checkpoints during fault recovery.
• Agentic multi-round trainingIt natively supports multi-turn interactions, tool calls, visual context carry-over, and loss masking, adapting to the "execution → observation → decision" closed loop.
• Algorithm suite supportBuilt-in GRPO, GSPO, SAPO and On-Policy Distillation algorithms provide rule-based rewards and GenRM generative evaluation.

The technical principles of Relax

• Service-oriented asynchronous architectureThe Actor, Critic, and Rollout roles are encapsulated as independent Ray Serve services, and the TransferQueue is used for streaming micro-batch transfers, which decouples computing resources and enables pipelined parallelism, eliminating the serial waiting bottleneck of the traditional Collocate mode.
• Full-modal data pipelineIt features a built-in Omni Processor for unified processing of text, image, audio, and video data, supports modality-aware parallel and end-to-end asynchronous pipelines, and allows independent reading and writing of different fields of the same sample through a Field-Level storage mechanism, adapting to the multi-stage computational characteristics of RL.
• Fault tolerance and resilience mechanismsIt adopts a two-level recovery strategy (stateless roles restart in place, stateful roles recover globally), and works with the DCS distributed checkpoint service to achieve direct weight synchronization between GPUs, avoiding disk I/O overhead, and supports NCCL/TCP dual-channel adaptation across cluster topologies.
• Asynchronous consistency controlThe On/Off-Policy mode can be flexibly controlled by the max_staleness parameter. StreamingDataLoader supports incremental consumption, and DCS asynchronous weight broadcasting overlaps with training computation, achieving a balance between throughput and policy freshness.

How to use Relax

• Environment DeploymentPull the official Docker image (pre-installed with CUDA, PyTorch, Megatron-LM, SGLang, Ray):docker pull relaxrl/relax:latestWhen starting the container, mount the GPU and the working directory.
• Data and Model Preparation:use hf download Download the datasets (such as dapo-math-17k, multimodal-open-r1-8k-verified) and the base models (Qwen3-4B, Qwen3-VL-4B, Qwen3-Omni-30B-A3B) to your local directory.
• Start training:set up EXP_DIR The environment variable points to the root directory of the data, and the corresponding modal bash script is executed:
  • Text task (8 cards):bash scripts/training/text/run-qwen3-4B-8xgpu.sh
  • Visual Language Task (8 cards):bash scripts/training/multimodal/run-qwen3-vl-4B-8xgpu.sh
  • Full-modal task (16 cards/dual machines):bash scripts/entrypoint/spmd-multinode.sh scripts/training/multimodal/run-qwen3-30B-A3B-omni-16xgpu.sh
• Model exportAfter training, use scripts/tools/convert_torch_dist_to_hf_bridge.py Convert Megatron DCP format weights to HuggingFace format.

Key information and usage requirements for Relax

• Open source main bodyThe RedAI Infra AI platform team of Xiaohongshu was officially open sourced on April 15, 2026.
• Product Positioning: A high-performance asynchronous reinforcement learning post-training engine for all-modal and agentic scenarios.
• Technology baseIt is built on the Megatron-LM training framework and the SGLang inference engine, and adopts the Ray Serve service architecture.
• Core ArchitectureThe six-layer service architecture (Entrypoints→Orchestration→Components→Engine→Backends→Distributed) uses TransferQueue to implement an asynchronous data bus and DCS to implement distributed weight synchronization.
• Performance indicatorsThe 16×H800 fully asynchronous mode is 76% faster than the Colocate baseline end-to-end speedup and 20% faster than veRL. The MoE model has an R3 overhead of only 1.9% (veRL has an R3 overhead of 32%).
• Modal supportIt natively supports unified training of text, images, audio, and video, and has been verified to achieve stable convergence of Qwen3-Omni-30B in four modes for 2000+ steps.
• Algorithm CoverageBuilt-in GRPO, GSPO, SAPO, and On-Policy Distillation, supporting rule-based rewards and GenRM generative evaluation.
• Hardware configurationA single node requires at least 8×H800 (4B model), and full-modal Omni training requires 16×H800 across 2 nodes. It supports elastic scaling and dynamic adjustment of Rollout resources.

Relax's core advantages

• Native support for all modesIt features a built-in unified processing pipeline for text, audio, and video, and is one of the few open-source frameworks that supports end-to-end post-training of Qwen3-Omni, with verified stable convergence over 2000 steps.
• Ultimate asynchronous performanceTransferQueue achieves complete decoupling of Rollout/Actor/Critic, and its 16×H800 fully asynchronous mode is 76% faster than Colocate and 20% faster than veRL.
• Service-oriented fault-tolerant architectureBased on Ray Serve's independent fault domain design, it supports minute-level automatic recovery and elastic scaling, and DCS achieves second-level weight synchronization without the need for disk rollback.
• Agentic native designBuilt-in multi-turn interaction, tool call, visual context carry-over and loss masking, adapted for closed-loop training of "execution → observation → decision".
• MoE High-Efficiency SupportR3 (Rollout Routing Replay) has an overhead of only 1.9%, achieving near-zero overhead replay compared to veRL's 34% performance loss.
• Production-level operations and maintenanceHealthManager provides automatic monitoring, Metrics supports multi-platform integration (WandB/TensorBoard), Apprise provides real-time alerts, and supports dynamic scaling via HTTP API.

Relax's project address

• GitHub repositoryhttps://github.com/redai-infra/Relax
• arXiv technical paperhttps://arxiv.org/pdf/2604.11554

Comparison of Relax's similar competing products

Application scenarios of Relax

• Post-training of large multi-modal modelsUnified optimization of Omni models that support text, image, audio, and video input (such as Qwen3-Omni) to enhance cross-modal understanding and generation capabilities.
• Agentic intelligent agent development: Train intelligent agents with multi-round tool calls, environmental interaction and autonomous decision-making capabilities to adapt to closed-loop tasks of "execution → observation → decision" (such as DeepEyes multi-round visual reasoning).
• Visual language task optimization: Perform RL post-training for tasks such as visual question answering (VQA), image description, and video understanding (NextQA) to improve the accuracy of multimodal reasoning.
• Mathematical and Code ReasoningEnhance the capabilities of large models in mathematical problem solving (DAPO-Math), logical reasoning, and code generation through algorithms such as GRPO/GSPO.
• Efficient Training of MoE ModelSupports low-cost R3 replay training for MoE architectures such as Qwen3-30B-A3B, reducing routing errors and computational overhead.