AB
AB
AiBoss

All Categories

project

autoresearch - Karpathy, an open-source AI framework for autonomous scientific research experiments.

Autoresearch is an open-source AI autonomous research experiment framework by Andrej Karpathy. The framework allows an AI agent to automatically run nanochat training experiments on a single GPU: autonomously tuning hyperparameters, executing training, analyzing and validating results...

What is AutoResearch?

Autoresearch is an open-source AI autonomous research experiment framework by Andrej Karpathy. The framework allows AI agents to automatically run nanochat training experiments on a single GPU, autonomously tuning parameters, executing training, analyzing and validating metrics, and deciding on the next optimization direction, completely automating the traditional manual cycle of parameter tuning, running experiments, reviewing logs, and then tuning again. Autoresearch has an extremely low barrier to entry, can run on a single GPU, and transforms AI from a mere object of research into an independent executor of research.

The main functions of autoresearch

  • Autonomous code iterationAI Agent Direct Editing train.py The file allows you to modify the entire training process configuration, including model architecture, optimizer, and hyperparameters.
  • Fixed-time budget experimentEach training session is strictly limited to 5 minutes of wall-clock time to ensure that experimental results are comparable across platforms.
  • Automatic Improved Filtering:use val_bpb(Verification bits per byte) is a unified metric; valid modifications that lower the metric are automatically retained, while invalid attempts are discarded.
  • Human-controlled research organizations:pass program.md Define agent behavior and research strategies, and iteratively optimize the "research organization code" instead of directly modifying the code.
  • Single-file focused designAgent only modified train.py A single file keeps the scope of the experiment under control and the diff is reviewable.

The technical principles of autoresearch

  • Simplified nanochat trainingA lightweight GPT model implemented on a single GPU, integrating the Muon optimizer and AdamW, supporting complete training loops.
  • BPE word segmentation and data flow:prepare.py Responsible for one-time data preparation and BPE segmenter training, providing standardized data loading and evaluation tools.
  • Unified evaluation indicatorsUsing vocab-size-independent val_bpb Metrics ensure that experimental results before and after the architecture change are fair and comparable.
  • Agent - Closed-loop environment:program.md As a "skill file" for the agent, the agent reads instructions → modifies code → runs training → evaluates metrics → decides to keep or discard → iterates in a loop.
  • Time standardization mechanismRegardless of changes in model size or batch size, training always runs for a fixed duration, eliminating the impact of hardware differences on experimental comparability.

How to use autoresearch

  • Install UV:implement curl -LsSf https://astral.sh/uv/install.sh | sh Install UV Project Manager.
  • Install dependenciesRun after entering the repository directory. uv sync Complete the installation of project dependencies.
  • Data preparation:implement uv run prepare.py Download the training data and train the BPE word segmenter (one-time, approximately 2 minutes).
  • Verification EnvironmentManual operation uv run train.py Confirm that a single training process is executed normally (approximately 5 minutes).
  • Start AgentLoad AI agents such as Claude/Codex into the IDE and point them to... program.md document.
  • Start researchInput prompt words such as “Hi have a look at program.md and let’s kick off a new experiment” Start automatic iteration.
  • View resultsCheck the experiment log the next day and train.py The modification history can be used to obtain the optimized model and improvement path.

The core advantages of autoresearch

  • Minimalist designOnly three core files (prepare.py,train.py,program.mdIt has zero external dependencies and can run on a single GPU.
  • Fair comparisonFixed 5-minute time budget and vocab-independent metrics ensure that any architectural changes can be evaluated on the same benchmark.
  • Human-machine collaborationHuman beings through program.md By setting research strategies, the Agent is responsible for executing code experiments, thus realizing the division of labor between humans setting directions and AI conducting experiments.
  • High throughputIt can perform approximately 12 experiments per hour and complete nearly 100 automatic iterations overnight, far exceeding the efficiency of manual experiments by humans.
  • AuditabilityThe single-file modification mode makes the diff of each experiment clear and traceable, making it easier to understand the agent's improvement path.

The project address for autoresearch

  • GitHub repositoryhttps://github.com/karpathy/autoresearch

Comparison of similar products from autoresearch

Comparison Dimensions autoresearch SciClaw
Product Positioning LLM Training Dedicated Autonomous Research Agent AI Colleagues in General Scientific Research / Full-Lifecycle Research Orchestration System
Core mission Automatically modify training code, adjust hyperparameters, run experiments, and select and improve. Literature review, experimental design, computational simulation, paper generation, and peer review responses.
Research subjects Nanochat model training (single GPU deep learning) Research tasks in multiple disciplines including life sciences, chemistry, physics, and materials.
Experiment Execution Directly modify the Python code and run the training loop Autonomously disassemble tasks, utilize scientific tools, run computational simulations, and dock with experimental equipment.
Evaluation mechanism Fixed 5-minute training + val_bpb Automatic filtering of metrics to keep/discard Experimental results analysis, paper quality assessment, hypothesis verification
knowledge base No, each experiment makes decisions independently based on code and metrics. Long-term memory project data, experimental records, literature, email history
Hardware Requirements Single NVIDIA GPU Runs in the cloud or deployed locally, supporting AI glasses and automated experimental equipment.
Interaction methods pass program.md Command file + AI agent for automatic execution Conversational interaction (Web/Lark/DingTalk/Telegram/Discord)
open source MIT is completely open source Closed-source product (internal testing requires an invitation code)
Target users AI researcher, LLM training engineer Researchers, doctoral students, and corporate R&D personnel
Output results Optimized model weights + experimental logs Paper drafts, PowerPoint presentations, posters, peer review responses, lab reports
Depth of autonomy Automatic code-level iteration, but limited to a single training task. Project-level full lifecycle orchestration supports backend experiments and self-repair that can take several days.

Application scenarios of autoresearch

  • Automatic Hyperparameter Search for LLM TrainingIt automatically explores combinations of learning rate, batch size, model depth, etc., to find the optimal configuration.
  • Architectural Innovation Validation: Quickly verify the practical effects of novel attention mechanisms, positional encoding, or optimizer variants.
  • Low-cost model tuningAutomatically optimizes the performance of small models in a single GPU environment, suitable for individual researchers or teams with limited resources.
  • Exploring AI Research MethodologiesAs a proof-of-concept platform for autonomous AI research, it explores the feasibility boundaries of "AI scientists".
Back