JoyAI-VL-Interaction - JD.com's open-source real-time video visual language interaction model - AiBoss

What is JoyAI-VL-Interaction?

JoyAI-VL-Interaction is an open-source real-time video visual language interaction model from JD.com's Joy Future Academy. It is the world's first full-stack open-source interaction model system. JoyAI-VL-Interaction enables large models to move from question-and-answer to simultaneous viewing and speaking, continuously observing the video stream, autonomously determining when to speak, and responding to changes in the visuals in real time. It also supports delegating complex tasks to a background agent. In 58 blind reviews by real users, it achieved a 77.6% win rate compared to Doubao Video Call Assistant and an 87.9% win rate compared to Gemini.

Main functions of JoyAI-VL-Interaction

Active visual responseIt continuously observes the camera/live stream/monitoring stream and autonomously determines when to speak and when to remain silent, without requiring users to ask questions in rounds.
Real-time streaming interactionIt responds instantly to the ongoing video stream, providing feedback within seconds when the scene changes, rather than summarizing the entire video afterward.
Agent delegationWhen encountering tasks such as code generation, tool invocation, or complex reasoning, the task is automatically handed over to the large backend model or Agent, while the frontend continues to observe.
Multimodal input/outputIt supports voice input/output, visual interface, and long-term memory. ASR/TTS/interface can be replaced as needed.
Long-term memoryIt possesses minute-level visual memory, allowing it to recall details of scenes from minutes ago and answer questions accurately.

The technical principle of JoyAI-VL-Interaction

Visual-first interaction decisionAt its core is a decision that is executed automatically every second—speak, remain silent, or delegate. The model is built on the JoyAI-VL-8B visual language instruction model, using speech as a pluggable input and output. The model's sole task is to observe the scene and determine the timing of the action.
Predictive video codingIt adopts the AdaCodec predictive video codec, which consumes only a small number of tokens for predictable frames, preserves complete details when the scene actually changes, and makes the token budget grow slowly over time rather than expanding frame by frame, supporting long-term real-time streaming processing.
Temporally aligned behavioral learningThe model is trained using over 4 million time-series interaction segments annotated second by second. Each data point is precisely labeled with when to speak, remain silent, or delegate, and is further fine-tuned using reinforcement learning to enable behavior to learn from the data.
Pluggable system architectureA complete deployment system is built around the core model, including streaming ASR/TTS, long-term memory modules, a visual UI, and a backend model bridge. All components are replaceable independently, and the entire system runs on standard vLLM infrastructure, gaining native day-0 support from vLLM-Omni.

How to use JoyAI-VL-Interaction

Cloning repositoryAccess GitHub open-source repositories jd-opensource/JoyAI-VL-Interaction Obtain the complete code and deployment system.
Get the modelDownload from Hugging Face jdopensource/JoyAI-VL-Interaction-Preview Model weights.
Prepare the environmentBased on standard vLLM infrastructure deployment, it supports various video inputs such as cameras, RTSP surveillance streams, and live streams.
Custom componentsReplace ASR, TTS, voice services, Agent, API, or front-end interface as needed to integrate with your own business systems.
Operating experienceOnce started, point the camera or live stream, and the model will enter a real-time observation and interaction state.

JoyAI-VL-Interaction's core advantages

Full-stack open sourceThe 8B model, training recipe, 4 million time series data points, and a complete deployable system are all open, reproducible, and scalable.
Live on-siteIt can continuously observe the live stream for several hours with a response latency of less than 1 second, truly being "present" rather than being polled and woken up.
Visual trigger initiativeIt can autonomously decide when to speak based on the content of the screen, and achieve zero-delay alarms in scenarios such as monitoring and early warning, and real-time translation.
front-end and back-end separationThe front-end model maintains uninterrupted visual observation, while the back-end processes complex tasks, and the results naturally return to the dialogue without interrupting the interaction flow.
Lightweight and deployableThe 8B parameter is compact and can run on standard infrastructure, lowering the barrier to deployment for real-time AI assistants.

JoyAI-VL-Interaction project address

Project official website: https://joyai-vl-video-future-academy-jd.github.io/JoyAI-VL-Interaction/
GitHub repository: https://github.com/jd-opensource/JoyAI-VL-Interaction/
HuggingFace model library: https://huggingface.co/datasets/jdopensource/JoyAI-VL-Interaction
Technical Papers: https://echovideo.jd.cn/JoyAI-VL-Interaction/JoyAI-VL-Interaction-Reportv1.pdf

Comparison of JoyAI-VL-Interaction with similar competing products

Comparison Dimensions	JoyAI-VL-Interaction	Doubao video call model
Model size	8B Parameters, Lightweight Visual-Language Interaction Model	Large parameter scale, general multimodal model
Core Paradigm	A vision-first proactive interaction model that autonomously decides "speak/remain silent/delegate" every second.	The polling question-answering model only processes the current frame after the user triggers the response.
Real-time processing	Continuous observation of streaming video, with real-time response to changes in the image.	Primarily handles static image snapshots taken at the moment a user asks a question.
Time Awareness	With built-in time awareness capabilities, it can perform time-sequenced tasks such as "remind in 20 seconds" and "broadcast every 3 seconds".	The timing task performed inconsistently and failed to provide timely reminders during testing.
Memory mechanism	Long-term visual memory lasting up to minutes allows users to recall details of images from several minutes ago.	Long-term visual memory is prone to errors, such as giving the wrong number of meatballs in a test.
Visual trigger	Voice output is automatically triggered by visual events, without requiring user input.	A response can only be generated if the user actively asks a question; it cannot generate alerts automatically.
Continuous tracking	Continuously track caption changes, repeat action counts, and app interface switching.	It is difficult to maintain cross-frame state, and counting and translation are prone to interruption.

Application scenarios of JoyAI-VL-Interaction

Security monitoringIt provides real-time monitoring of the stream and instant voice alerts for abnormal events (such as falls or intrusions), eliminating the need for manual screen monitoring.
Elderly/ChildcareContinuously observe home footage and proactively remind individuals to avoid dangerous behaviors (such as approaching the stove or going out alone).
Live Stream Guide / E-commerce Shopping GuideIt provides real-time commentary on the live stream, automatically introduces product details, or proactively offers styling suggestions based on the user's outfit.
Real-time translationWhen watching foreign language videos or communicating face-to-face, it continuously recognizes subtitles/dialogues and provides real-time oral translation.
Operating Instructions: Guide users to operate the app or device by providing step-by-step prompts that follow screen changes, rather than simply describing static screenshots.
AI Glasses/Accessibility AssistanceAs the core visual brain of AI glasses, it describes the surrounding environment for visually impaired people in real time and proactively alerts them to obstacles.