Tarsier2 - ByteDance's Large-Scale Visual Understanding Model

What is Tarsier2?

Tarsier2, developed by ByteDance, is an advanced large-scale visual language model (LVLM) that generates detailed and accurate video descriptions, demonstrating outstanding performance across various video understanding tasks. The model achieves performance improvements through three key upgrades: expanding the pre-training data from 11 million to 40 million video-text pairs, enriching the data volume and diversity; performing fine-grained temporal alignment during supervised fine-tuning; and automatically constructing preference data based on model sampling, applying Direct Preference Optimization (DPO) training. In the DREAM-1K benchmark, Tarsier2-7B outperforms GPT-4o by 2.8% and Gemini-1.5-Pro by 5.8%. It also achieves new state-of-the-art results on 15 public benchmarks, covering tasks such as video question answering, video localization, illusion testing, and embodied question answering.

Tarsier2's main functions

• Detailed video descriptionTarsier2 can generate high-quality video descriptions that cover various details in the video, including actions, scenes, and plot.
• Video Q&AThey can answer specific questions about the video, demonstrating strong spatial and temporal understanding and reasoning abilities.
• Video positioningTarsier2 can detect and locate the occurrence time of specific events in videos, and supports the location of multiple video segments.
• Hallucination TestBy optimizing the training strategy, Tarsier2 significantly reduces the likelihood of the model generating false information.
• Multilingual supportIt supports video description generation in multiple languages, further expanding its application scenarios.

Technical principles of Tarsier2

• Large-scale pre-training dataTarsier2 expands the pre-training data from 11 million to 40 million video-text pairs, increasing the scale and diversity of the data. The data includes short videos from the internet, narration videos from movies or TV series, and video descriptions and question-answer pairs automatically generated through multimodal LLM.
• Supervised fine-tuning of fine-grained time alignment (SFT)During the supervised fine-tuning phase, Tarsier2 introduced 150,000 finely annotated video descriptions, each corresponding to a specific timestamp. This time-aligned training significantly improved the model's accuracy and detail capture capabilities in video description tasks, while reducing the generation of illusions.
• Direct Preference Optimization (DPO)Tarsier2 automatically constructs preference data through model sampling and applies Direct Preference Optimization (DPO) for training. Based on the positive and negative sample pairs generated by the model, the generation quality of the model is further optimized to ensure that the generated video descriptions are more in line with human preferences.

Tarsier2's project address

• GitHub repository:https://github.com/bytedance/tarsier
• arXiv technical paper:https://arxiv.org/pdf/2501.07888

Application scenarios of Tarsier2

• Video DescriptionTarsier2 can generate high-quality, detailed video descriptions that cover various details in the video, including actions, scenes, and plot.
• Hallucination TestTarsier2 excels at reducing the generation of illusions. Through training with direct preference optimization (DPO) and fine-grained temporal alignment, Tarsier2 significantly reduces the likelihood of generating false information.
• Multilingual supportTarsier2 supports multilingual video description generation, meeting the needs of different language environments.
• Embodied QuestionsTarsier2 also performs well in embodied question answering (Embodied QA) tasks, combining visual and linguistic information to provide accurate guidance to embodied agents.