FutureX - A dynamic real-time evaluation benchmark launched by ByteDance in collaboration with universities such as Fudan University

What is FutureX?

FutureX, jointly released by research teams from ByteDance, Fudan University, Stanford University, and Princeton University, is a dynamic, real-time evaluation benchmark designed specifically for LLM agents' future prediction tasks. It collects future event questions in real-time from 195 high-quality websites through a semi-automated pipeline, automatically acquiring and scoring the actual results after the events are resolved, effectively avoiding data contamination. FutureX covers multiple fields including politics, economics, finance, sports, and entertainment, and includes various question types such as single-choice, multiple-choice, open-ended ranking, and numerical prediction, divided into four difficulty levels to comprehensively evaluate the reasoning and predictive capabilities of LLM agents.

FutureX's main functions

• Dynamic real-time updatesFutureX can collect future event issues in real time and automatically obtain the actual results for scoring after the events are resolved, ensuring the timeliness and dynamism of the evaluation.
• Avoid data pollutionBy focusing on predicting future events, FutureX ensures that the answer has not yet occurred when the agent makes the prediction, thus avoiding data contamination and guaranteeing the fairness of the evaluation.
• Simulate real-world challengesFutureX places LLM agents within the flow of information in the real world, requiring them to predict future events. This necessitates that the agents possess advanced cognitive skills such as information gathering, data synthesis, probability trade-offs, and causal reasoning.
• Large-scale cross-domain coverageFutureX collects questions from 195 high-quality websites, covering multiple fields such as politics, economics, finance, sports, and entertainment, providing a comprehensive evaluation environment.
• Automated evaluation processFutureX's assessment process is fully automated, automatically updating questions, collecting answers, and providing objective scores daily, improving the efficiency and scalability of assessments.
• Multiple types of questions and difficulty levelsFutureX includes various question types such as single choice, multiple choice, open ranking, and numerical prediction, and is divided into four difficulty levels to comprehensively evaluate the capabilities of LLM agents.
• Promote the development of LLM agenciesFutureX provides a dynamic, pollution-free evaluation standard for LLM agents, pushing them towards the level of professional human analysts and improving their performance in complex reasoning and prediction tasks.

FutureX's core advantages

• Design PrinciplesFutureX aims to provide a dynamic, comprehensive, and data-free assessment that simulates real-world challenges to evaluate the core intelligence of LLM agents.
• No data pollutionFutureX avoids data pollution by focusing on predicting future events, ensuring that the answer has not yet occurred when the agent makes the prediction.
• Simulate real-world challengesFutureX places agents within the flow of information in the real world, requiring them to predict future events. This necessitates that agents possess advanced cognitive skills such as information gathering, data synthesis, probability trade-offs, and causal reasoning.
• Large-scale and cross-domain coverageFutureX collects questions from 195 high-quality websites through a semi-automated pipeline, covering multiple fields such as politics, economics, finance, sports, and entertainment.
• Dynamic and automated evaluation processFutureX automatically updates questions, collects answers, and provides objective scoring daily to ensure the timeliness, objectivity, and scalability of the assessment.

The construction process of FutureX

• Website collection and filteringWe used an AIME proxy to collect a large number of relevant website URLs, and then used LLM and manual review to select high-quality websites, ultimately identifying 195 as the event database.
• Event template generationCreate event templates for each website; these templates can generate events that adapt to different times based on variables.
• Event planning: Generate prediction questions daily from the event database, including manipulation of events (such as adding random options) and filtering (removing harmful, subjective, or overly simplistic events).
• Agency forecasting and evaluationThe agent model is triggered daily to predict new events and automatically obtains the actual results for scoring after the events are resolved.
• Continuous updates and maintenanceThe event database is updated daily, removing events with unavailable results and adding new events to ensure the dynamic nature and timeliness of the benchmark.

FutureX's data characteristics

• Real-timeFutureX's data is updated in real time, collecting questions about future events from 195 high-quality websites daily to ensure that the assessment content is in sync with current information.
• diversityThe data covers multiple fields such as politics, economics, finance, sports, and entertainment, and includes various types of questions such as single choice, multiple choice, open ranking, and numerical prediction.
• Pollution-freeIt focuses on predicting future events, ensuring that the answer has not yet occurred when the proxy is making the prediction, thus avoiding data contamination and guaranteeing the fairness of the assessment.
• DynamismFutureX's events and answers are dynamically updated. The event database adds new events or removes unavailable events based on the actual situation, keeping the data dynamic.
• ChallengingBy using event filtering and difficulty levels, FutureX ensures the challenge of the questions, ranging from simple multiple-choice questions to complex open-ended questions, to comprehensively evaluate the capabilities of LLM agents.
• large-scaleFutureX is currently the largest and most diverse real-time future prediction benchmark, generating approximately 500 events per week, providing a rich sample for evaluation.
• reliabilityThrough rigorous data screening and manual review, we ensure the reliability and quality of data sources, providing a credible basis for the assessment.

FutureX project address

• arXiv technical paper: https://arxiv.org/pdf/2508.11987

FutureX's experimental results

• Overall ResultsGrok-4 and Gemini-2.5-flash Deep Research performed best on the most difficult tasks, while basic LLM performed well on simple tasks.
• Results at different difficulty levelsAs the difficulty of the task increases, the model performance drops significantly, especially at Level 4 (super agent level), where the model struggles the most.
• Results from different fieldsDifferent models exhibit different advantages in different fields. For example, the GPT model performs well in the fields of cryptocurrency and technology, while DouBao-Seed1.6-Thinking performs well in the fields of finance and economics.
• Factor analysisLinear regression analysis revealed that difficulty level, domain, and model name have a significant impact on performance.
• Case StudiesThis includes a comparison between LLM agents and Wall Street financial analysts, the impact of fake websites on agents, and an assessment of real-time search capabilities.

FutureX application scenarios

• Financial sectorFutureX can be used to evaluate the ability of LLM agents to predict future events such as stock prices and economic indicators, helping financial institutions screen high-performance analytics agents.
• Policy makingIt provides policymakers with reliable intelligent agent assessment tools to help them assess the potential impact of different policies.
• Business DecisionsIt helps businesses assess market trends and consumer behavior, providing support for business decisions.
• Technical Trend AnalysisIt predicts technological development and innovation trends, providing decision-making support for technology companies and investors.
• Sports event predictionsIt predicts the results of sports competitions and the performance of athletes, providing a reference for sports betting and event organizers.
• Entertainment industryIt predicts the popularity and box office revenue of entertainment products such as movies and music, providing support for decision-making in the entertainment industry.