JoyAI-Video-Edit - JD.com's open-source real-time streaming video editing model - AiBoss

What is JoyAI-Video-Edit?

JoyAI-Video-Edit is a real-time streaming video editing model developed and open-sourced by JD.com. Based on a 16-parameter autoregressive diffusion architecture, it achieves an inference speed of 30FPS at 720P resolution and supports stable streaming editing of any duration. Users can modify people, scenes, styles, and objects in real time during video playback using natural language commands, without waiting for the complete video to be generated. The model outperforms all streaming editing methods in the OpenVE-Bench benchmark, leading in all metrics, and also provides a large-scale data synthesis path for embodied intelligence.

Main functions of JoyAI-Video-Edit

Real-time streaming editingVideo frames can be edited instantly upon arrival, without needing to know the complete sequence in advance.
Open instruction controlIt supports global style migration, adding, deleting, and modifying local objects, background replacement, action adjustment, and reference image-guided editing.
Arbitrary duration processingBreaking through the limitations of seconds or minutes, it allows for stable editing while the video is playing.
720P high throughput deploymentThe end-to-end pipeline achieves 30.19 FPS at 720×1280 resolution.
Multi-dimensional editing capabilitiesIt covers tasks such as character costume changes, scene transformations, style conversions, object replacements, and subtitle editing.

The technical principles of JoyAI-Video-Edit

MLLM condition encoder: Encode natural language instructions into editing conditions using a multimodal large language model to achieve open semantic control.
Causal Video VAEA variational autoencoder with a causal temporal structure is used to compress and reconstruct video frames, ensuring temporal consistency in streaming scenarios.
16B parameter MMDiT backboneUsing the 16B multimodal diffusion Transformer as the core, autoregressive diffusion generation is performed by fusing textual and visual features.
Autoregressive distribution matching distillationThe distillation strategy, which aligns with the autoregressive distribution, accelerates inference and significantly reduces the distributional discrepancy between training and inference.
Long-term optimization and bounded key-value inferenceThe system introduces long-term stability optimization and bounded key-value caching mechanism to suppress cumulative timing drift and maintain high throughput.

How to use JoyAI-Video-Edit

Environmental preparationCreating a Conda environmentpython=3.10And installrequirements.txtrely.
Download weight: Obtain model files from Hugging Face and place checkpoints according to the specified directory structure.
Start service:EnterdeployDirectory Executionbash run_server.shStart the local server.
Access interfaceOpen in browserhttp://localhost:8080Enter the interactive editing interface.
Real-time editingUpload videos or access camera streams, and input natural language commands to edit while playing.

JoyAI-Video-Edit's core advantages

Real-time streaming editingVideo frames are processed as soon as they arrive, without waiting for the complete sequence, achieving a true "play and edit" interactive experience.
Excellent in both speed and qualityAt 720P resolution, end-to-end inference reaches 30.19 FPS, surpassing all streaming and offline editing methods in the OpenVE-Bench benchmark.
Stable output for any durationBreaking through the limitations of traditional models that only support second-level or minute-level segments, it can continuously perform drift-free and stable editing while the video is playing.
Open semantic controlBased on the MLLM conditional encoder, it supports multi-dimensional editing driven by natural language instructions, including global style, local objects, background replacement, and reference image guidance.
High-efficiency inference architectureThe 16B-parameter autoregressive diffusion Transformer, combined with distribution matching distillation and bounded KV caching, significantly reduces the training-inference discrepancy and suppresses temporal cumulative error.

JoyAI-Video-Edit project address

GitHub repository：https://github.com/jd-opensource/JoyAI-Video-Edit
HuggingFace model library：https://huggingface.co/jdopensource/JoyAI-Video-Edit
arXiv technical paper：https://arxiv.org/pdf/2608.03974

Comparison of JoyAI-Video-Edit with similar competitors

Dimension	JoyAI-Video-Edit	SANA-Streaming
Model size	16B parameters	2B parameters
Reasoning speed	37.21 FPS (DiT)	Approximately 54 FPS (DiT)
Editing quality	3.60 (OpenVE-Bench)	Approximately 2.6
Editing type	Open-ended multi-dimensional editing	Streaming editing
Resolution support	720P stable output	Mainly lower resolution
Timing stability	Bounded key-value optimization for arbitrary duration	Short clips

Application Scenarios of JoyAI-Video-Edit

Short video creationCreators can change character costumes and adjust scene styles in real time during video playback, improving content production efficiency.
Live Streaming Real-Time EffectsThe host can change the background or add virtual elements while broadcasting to enhance the visual impact of the interactive live stream.
Home decoration and interior designUsers can change furniture, wall materials, and lighting effects in real time while browsing room videos, assisting in decorating decisions.
Film and television post-production previewDuring the editing stage, the director quickly tries different visual styles and scene replacements to reduce the cost of trial and error.
Embodied Intelligent Data SynthesisConverting human hand operation videos into robotic arm operation materials allows for low-cost expansion of robot training data.