VeOmni - ByteDance's open-source full-modal PyTorch native training framework

What is VeOmni?

VeOmni is an open-source, multimodal distributed training framework developed by ByteDance's Seed team, based on PyTorch. VeOmni is model-centric, decoupling distributed parallel logic from model computation. It supports flexible combinations of various parallel strategies (such as FSDP, SP, EP) and can efficiently scale to ultra-long sequences and large-scale MoE models. VeOmni provides a lightweight multimodal interface, simplifying the integration of multimodal encoders and decoders. It integrates optimization techniques such as dynamic batching and efficient operators, significantly improving training efficiency and stability. VeOmni has been applied in several cutting-edge projects, contributing to the research and development of large multimodal models.

VeOmni's main functions

• Supports full-modal model trainingVeOmni supports model training in any modality (such as text, image, audio, video, etc.) and is suitable for a variety of tasks from single-modal to multi-modal.
• High-efficiency distributed trainingIt supports flexible combinations of various parallel strategies (such as FSDP, SP, EP) and can be efficiently scaled to large-scale GPU clusters.
• Support for very long sequencesIt supports training of ultra-long sequences up to 192K, making it suitable for processing complex modal data such as high-definition images and long videos.
• Lightweight interface and ease of useSupports rapid integration of multimodal codecs, simplifying the model development process.
• System-level optimizationIt integrates technologies such as dynamic batch processing, high-efficiency operators, recomputation and memory optimization, and ByteCheckpoint to improve training efficiency and stability.
• Training stabilityIt performs well in complex multimodal tasks, supports stable convergence, and is suitable for a variety of scenarios in practical applications.
• Flexible model extensionIt supports multiple model architectures (such as MoE, Transformer, etc.) and allows users to customize model components to meet different research and development needs.

VeOmni's technical principles

• Model and system decouplingVeOmni separates model definition from distributed training logic, completely decoupling model code from parallel strategies. Users can configure parallel strategies through a high-level API without modifying the model code.
• Distributed parallel strategyThis approach significantly reduces the memory requirements of a single GPU by sharding model parameters, gradients, and optimizer states across multiple devices. It supports training of ultra-long sequences by splitting activation tensors and optimizing communication. Furthermore, it improves the training efficiency of MoE models by sharding experts across multiple devices. Finally, it simplifies the management of n-D parallel strategies by designing parallel_state based on DeviceMesh, allowing for flexible combinations of different parallel strategies.
• Lightweight multimodal interfaceUsing the HuggingFace interface specification, users can quickly integrate multimodal codecs by implementing unified interface functions (such as lm_encode, lm_generate, etc.).
• System-level optimizationIt integrates multiple optimization techniques, such as dynamic batch processing, high-efficiency operators, recomputation and memory optimization, as well as ByteCheckpoint, to comprehensively improve training efficiency and stability.

VeOmni's project address

• GitHub repositoryhttps://github.com/ByteDance-Seed/VeOmni
• arXiv technical paper: https://arxiv.org/pdf/2508.02317

VeOmni application scenarios

• Multimodal content generationIt can generate images or videos based on text descriptions, or generate descriptive text for images or videos, and is widely used in creative design and content creation.
• Multimodal understanding and question answeringIt combines images and text to answer visual questions or handle complex question-answering tasks involving multiple modalities, thereby enhancing the intelligent interactive experience.
• Multimodal intelligent agentsIt supports the development of virtual assistants and multimodal robots that can interact with users and perform tasks through voice, text, and visual information.
• Content creation and editingGenerate creative design elements based on text descriptions to assist in content review and improve the efficiency of content creation and editing.
• Education and TrainingIt provides virtual training to enhance the interactivity and effectiveness of education and training.