Lance - ByteDance's open-source lightweight native unified multimodal model - AiBoss

What is Lance?

Lance is a lightweight, native unified multimodal model open-sourced by ByteDance's intelligent creation team. It uses only 3B activation parameters and supports the entire chain of tasks—image and video understanding, generation, and editing—within a single framework. The model is trained from scratch using a phased, multi-task approach, consuming only 128 A100 GPUs throughout the entire training cycle. It performs excellently in multiple benchmark tests, including GenEval and VBench, and is licensed under the Apache-2.0 open-source license, supporting commercial use.

Lance's main functions

Image understanding: Perform semantic parsing, content recognition, and visual question answering on the input image.
Image generationGenerates high-quality images based on text prompts, supporting complex compositions and attribute binding.
Image editingIt supports command-level editing such as background replacement, object addition and deletion, style conversion, and appearance reshaping.
Video UnderstandingPerform temporal analysis, action recognition, and semantic understanding on video content.
Video generationGenerates coherent videos based on text descriptions, supporting character movement and scene construction.
Video editingEnables single-step and combined video editing, including background transformation, subject replacement, and motion modification.
Multi-round consistency editing: Perform multiple rounds of editing on the same subject to maintain consistency in identity and style.

Lance's technical principles

Dual-stream hybrid expert architectureWhile sharing multimodal sequence representations, independent expert paths are assigned to the understanding and generation tasks to avoid interference between heterogeneous objectives.
Unified Interleaved Sequence RepresentationIt organizes text tokens, ViT semantic tokens, clean VAE latent tokens, and noisy VAE latent tokens into a unified sequence, supporting understanding, generation, and conditional editing.
Generalized 3D causal attentionThe sequence is segmented into modalities, and text tokens use causal attention while visual tokens use bidirectional attention, thus unifying the processing of multimodal understanding and generation.
Modality-aware position coding: Introducing rotational position encoding for heterogeneous visual tokens in images and videos to reduce signal interference between different modalities.
Phased multi-task trainingIt employs a progressive formula of pre-training, continuous training, and supervised fine-tuning to achieve multi-task collaboration within a limited computing power budget.

How to use Lance

Environmental preparationEnsure you have a Python environment and at least one CUDA-enabled GPU, either locally or in the cloud.
Cloning repository:implement git clone https://github.com/bytedance/Lance.git Download the project source code.
Install dependenciesRun after entering the project directory. pip install -r requirements.txt Install the necessary libraries.
Download weightGet the Lance pre-trained model weights from Hugging Face or the project Release page.
Running inferenceLoad the model based on the official example script, and use the input text or visual cues to generate, edit, or understand the task.

Lance's core advantages

Ultra-lightweightOnly 3B activation parameters are used, achieving the best balance between parameter quantity and performance in a unified model.
End-to-end unifiedA single model covers six major tasks: image and video understanding, generation, and editing, without the need to switch to a dedicated model.
Low-cost trainingTraining was completed from scratch using 128 A100 GPUs within budget, significantly lowering the barrier to reproduction.
Business friendlyThe Apache 2.0 open-source license allows for free commercial use, modification, and distribution.
Leading performanceIt outperforms existing open-source unified models in multiple benchmarks such as GenEval, GEdit-Bench, and VVBench.

Lance's project address

Project official websitehttps://lance-project.github.io/
GitHub repositoryhttps://github.com/bytedance/Lance
HuggingFace model libraryhttps://huggingface.co/bytedance-research/Lance
arXiv technical paper: https://arxiv.org/pdf/2605.18678

Comparison of Lance with similar competing products

Comparison Dimensions	Lance	TUNA	Show-o2
Activation parameter quantity	3B	7B	7B
Task Coverage	Image/video understanding, generation, and editing	Image/video understanding and generation	Image/video understanding and generation
Open source license	Apache-2.0	Unclear	Apache-2.0
GenEval Total Score	0.90	0.90	0.76
GEdit-Bench average score	7.30	6.52	Not listed
VBench Total Score	85.11	Not listed	Not listed
Architectural features	Dual-stream MoE + Modality-aware position coding	Unified Autoregressive Architecture	Unified Autoregressive Architecture

Lance's application scenarios

Intelligent content creationIt provides designers and creators with a one-stop image and video generation and editing tool to accelerate the realization of creative ideas.
Short video productionIt supports the rapid generation and editing of short video content, reducing the threshold and cost of video production.
Advertising and marketing materialsBatch generation of product images, posters, and promotional videos enables efficient iteration of visual materials.
Visual search enhancementEnhance the visual retrieval experience of e-commerce and content platforms by leveraging image and video understanding capabilities.
Educational content productionIt automatically generates teaching illustrations and demonstration videos, enriching the multimedia resources for online education and training.