CAR - An adaptive inference framework jointly launched by ByteDance and Fudan University
CAR (Certainty-based Adaptive Reasoning) is an adaptive reasoning framework jointly developed by ByteDance and Fudan University. It improves the performance of Large Language Models (LLM) and Multimodal Large Language Models (MLLM) across various tasks...
What is CAR?
CAR (Certainty-based Adaptive Reasoning) is an adaptive reasoning framework developed by ByteDance in collaboration with Fudan University. It improves the reasoning efficiency and accuracy of Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) across various tasks. The framework dynamically switches between short-form and long-form reasoning, determining whether detailed reasoning is needed based on the model's confidence in the answer (measured by perplexity level, PPL). When the model's confidence in a short answer is low, CAR triggers long-form reasoning to improve accuracy; when confidence is high, it directly outputs the short answer, saving computational resources and time. CAR performs exceptionally well in tasks such as Visual Question Answering (VQA) and Key Information Extraction (KIE), and demonstrates good performance in complex reasoning tasks such as mathematics.
Main functions of CAR
- Dynamic inference switchingCAR can intelligently switch between short answers and long-form reasoning. For simple questions, it directly outputs short answers to improve efficiency; for complex questions, it triggers long-form reasoning to ensure accuracy.
- Improve reasoning efficiencyCAR significantly reduces the number of tokens generated by the model, lowers computational costs and inference time, and improves the efficiency of the model in practical applications.
- Improve reasoning accuracyWhen detailed reasoning is required, activate long-form reasoning to improve the model's performance in complex tasks and ensure the accuracy of the reasoning results.
- Adaptable to multiple tasksCAR is applicable to tasks such as visual question answering (VQA) and key information extraction (KIE), and plays a role in complex tasks such as mathematical reasoning and common sense reasoning, and has a wide range of applicability.
CAR technology principles
- Model trainingThe LLM or MLLM is trained using training data containing both short and long-form reasoning answers. During training, the model learns how to generate appropriate short or long-form reasoning answers based on different prompts. The optimization objective is cross-entropy loss, which trains the model by minimizing the difference between the probability distribution of predicted tokens and the distribution of true tokens.
- PPL for getting short answersPerform short answer inference on the training data and calculate the perplexity level (PPL) for each short answer. PPL is a metric that measures the model's confidence in the answer; the lower the PPL, the higher the model's confidence in the answer.
- Gaussian distribution modelingAssume that the PPL scores for correct and incorrect short answers follow Gaussian distributions. Estimate the parameters of the distributions based on the training data, and calculate the mean and variance of the PPL for correct and incorrect answers, respectively, to obtain two Gaussian distribution models.
- Reasoning processFor a new input, first generate a short answer and calculate the Probability of Correct Answer (PPL). Then, calculate the probability of the correct answer at that PPL value based on a Gaussian distribution model.
CAR project address
- arXiv technical paper:https://arxiv.org/pdf/2505.15154
CAR application scenarios
- Visual Question Answering (VQA)CAR outputs short answers directly for simple questions, improving efficiency; and triggers long-form reasoning for complex questions, ensuring accuracy.
- Key Information Extraction (KIE)The inference method is dynamically selected based on confidence level, reducing the amount of tokens used and maintaining high accuracy.
- Mathematical reasoningIt directly outputs answers to simple math problems and performs detailed reasoning for complex problems, improving accuracy and efficiency.
- Common sense reasoningFor simple common-sense questions, the answer is given directly; for complex questions, long-form reasoning is used to reduce token consumption.
- Multimodal tasksBy combining textual and image information, and dynamically selecting reasoning methods, the accuracy and efficiency of multimodal tasks can be improved.