EdgeBench - ByteDance's AI learning capability benchmark framework

What is EdgeBench?

EdgeBench is a benchmarking framework launched by ByteDance's Seed team to evaluate the long-term learning capabilities of autonomous AI agents in real-world environments. The framework includes 134 real-world tasks covering six major areas, including scientific computing, software engineering, and combinatorial optimization. Through continuous operation for 12-72 hours, it tracks the agent's trial-observation-absorption-improvement learning curve, revealing the predictable scaling patterns of AI learning from its environment.

EdgeBench's main functions

• Long-term learning assessmentSimulate the Agent to run continuously in a real environment for 12-72 hours to track the complete learning curve.
• Multi-domain task coverageIt covers 134 real-world missions, spanning 6 major categories including science, engineering, and optimization.
• Pollution-proof design51 tasks are made public, and 83 tasks are reserved to prevent benchmark contamination and overfitting.
• Quantifying learning patternsWe discovered that Agent performance follows a log-sigmoid scaling pattern, with the learning speed roughly doubling every 3 months.
• Human benchmark comparisonIt provides a benchmark of 57.2 hours of completion time for experts and humans, quantifying the gap between humans and machines.

The technical principles of EdgeBench

• Environment Interaction Learning CycleEdgeBench constructs a closed-loop evaluation framework of trial-observation-absorption-improvement. The agent performs actions in a real task environment, receives environmental feedback, updates its strategy, and tries again, simulating the progressive learning process of humans in complex tasks.
• Time-segmented performance tracingThe system divides long-term operation into multiple stages, continuously records the Agent's performance scores at each time point, and forms quantifiable learning curve data, supporting detailed analysis of long-term learning dynamics.
• Cross-domain task modelingA unified assessment protocol was designed for six different types of tasks with varying cognitive difficulty, ranging from scientific computing to formal mathematics, to ensure that the assessment framework covers a wide range of real-world challenges.

Follow us on WeChat and reply with "open source",join inAI open source project discussion group

How to use EdgeBench

• Access the repositoryGo to GitHub to search ByteDance-Seed/EdgeBench Get the leaderboard and public task list.
• Select taskSelect target domain tasks from 51 publicly available tasks and understand their evaluation indicators and environmental configurations.
• Deploy AgentConnect the AI Agent to be tested to the task environment and configure it to run continuously for more than 12 hours.
• Data collectionRecord the Agent's performance scores at each time stage and generate learning curve data.
• Submit an evaluationThe results were compared and analyzed with models such as Claude Opus 4.8 and GPT-5.5 in the rankings.

EdgeBench's core advantages

• Real-world orientation: Evaluate the Agent's actual working capabilities based on real-world tasks rather than static question answering.
• Long-term dynamic trackingBreaking through the limitations of single-inference evaluation, it captures the continuous improvement trajectory of the agent during long-term operation.
• Predictable patternsWe found that the AI learning curve follows a highly predictable log-sigmoid scaling relationship (R² = 0.998).
• Anti-pollution mechanismThe 83 retained tasks effectively prevent the model from over-optimizing against the benchmark.
• Frontier Model CoverageTop models such as Claude Opus 4.8, GPT-5.5, Gemini 2.5 Pro, o3, and DeepSeek-V4-Pro have been evaluated.

EdgeBench project address

• Project official websitehttps://edge-bench.org/
• GitHub repositoryhttps://github.com/ByteDance-Seed/EdgeBench
• HuggingFace model libraryhttps://huggingface.co/datasets/ByteDance-Seed/EdgeBench
• Technical Papershttps://edge-bench.org/paper.pdf

EdgeBench's Competitive Comparison

Application scenarios of EdgeBench

• General Intelligence ResearchProvides a quantitative assessment standard for long-term learning capabilities for general intelligent initiatives such as "Seed Edge".
• Agent capability iterationIt helps developers identify bottlenecks in agents during long-running tasks and guides the direction of model optimization.
• Model Selection ReferenceThe study compares the long-term learning performance of models such as Claude, GPT, and Gemini across various fields using leaderboards.
• Human-machine capability benchmarkingUsing the benchmark of 57.2 hours for expert humans as a reference, we measure the progress of AI in approaching human level.
• Educational training designIt provides a standardized evaluation environment for research on AI autonomous learning and continuous algorithm improvement.