DataChef - An AI data recipe generation model open-sourced by Shanghai AI Lab in collaboration with Fudan University.

What is DataChef?

DataChef is an open-source AI data recipe generation model jointly developed by the Shanghai Artificial Intelligence Laboratory and Fudan University. The model automatically generates a complete data processing pipeline for large-scale model-adapted tasks through reinforcement learning, including executable code for steps such as data selection, cleaning, synthesis, and formulation. DataChef only requires the target task and available data sources as input; the AI can automatically "refine" the data. The 32-bit version has achieved performance close to Gemini-3-Pro in tests across six domains, including mathematics, coding, and finance, and surpasses industrial-grade expert recipes on some tasks, marking a shift in data engineering from manual experience to a new paradigm of automation.

DataChef's main functions

• Automatic recipe generationDataChef can automatically generate complete data recipes based on the target task and available data sources.
• Code outputThe model outputs executable Python data processing code, building an automated pipeline from raw data to the training set.
• Multi-stage processingIt supports various processing operations such as data selection, cleaning, enhancement, mixing, and deduplication.
• Immediate training and applicationThe generated data formulas can be directly used for fine-tuning large models to adapt them to specific domains.
• Quality verificationDataChef has a built-in data validator that can assess data quality and guide the optimization process.

DataChef's technical principles

• Task modelingDataChef models data recipe generation as an end-to-end reinforcement learning task. The policy model receives task instructions and generates data recipes containing natural language plans and Python code.
• Agent reward mechanismTo address the issue of high training feedback costs, the research team designed the Data Verifier as a proxy reward mechanism. The verifier achieves low-cost, real-time prediction of data quality by classifying samples into five quality levels and scoring based on sample subsets.
• Training optimizationThe training process employs cold-start supervised fine-tuning initialization, combined with the GRPO algorithm for online reinforcement learning optimization, and improves policy stability by decoupling inference and code generation, enabling the model to efficiently explore the optimal data scheme in a vast code combination space.

Key information and usage requirements for DataChef

• R&D teamShanghai Artificial Intelligence Laboratory (Shusheng Puyu Team) in collaboration with Fudan University
• Model size: Open source 32B parameter version (DataChef-32B)
• performance levelIt approaches the top-level closed-source model Gemini-3-Pro on 6 held-out test tasks, and surpasses industrial-grade expert formulations on some tasks.
• Core InnovationThis is the first time that end-to-end data recipe generation has been modeled as a global decision problem, and automatic optimization closure is achieved through online reinforcement learning.
• Data FoundationCovering 19 domains, 31 evaluation sets, and 257 source datasets.
• Environment configurationInstalling dependencies via pip in Python 3.12
• Hardware resourcesRunning a 32B model requires sufficient video memory or API access capabilities.
• Model endpoints: Requires configuring the API endpoints for the Planner and Coder models (supports OpenAI-compatible interfaces).

DataChef's core advantages

• End-to-end automationBreaking away from traditional local heuristic rules, it elevates data recipe generation to an end-to-end task, with AI directly outputting a complete and executable data processing pipeline without the need for manual arrangement of operation steps.
• Reinforcement learning drivenBy forming an automatic optimization loop through online reinforcement learning, the model can evolve itself in a vast code combination space, continuously explore better data solutions, and eliminate the need for repeated manual trial and error.
• Low-cost verification mechanismThe innovative model proposes the Data Verifier agent reward signal, which can predict data quality in real time without the need for complete model training, significantly reducing the cost of validating the effects of traditional data engineering.
• Superior performanceThe 32B parameter open-source model approaches Gemini-3-Pro in multiple tests and surpasses industrial-grade expert recipes in some complex tasks, proving that AI-generated solutions are superior to human experience-based designs.
• Open infrastructureIt builds a large-scale task pool covering 19 fields and is fully open source, providing systematic support for cutting-edge research such as automated data engineering and self-evolving AI.

How to use DataChef

• Environmental installationCreate a Python 3.12 virtual environment by...pip install -e .The command completes the DataChef installation.
• configuration fileCopy the example configuration file and name it...datachef.config.jsonEnter the model endpoint address, API key, and Planner and Coder model names.
• Ready to inputPrepare task files in JSONL format, with each line containing the task ID, task description and evaluation benchmark, and a list of candidate data sources.
• Run to generate:implementdatachef-eval --config testThe command initiates the process, and the system automatically generates a data recipe plan, Python code, and executes verification.
• Get output:existdata/code/anddata/data-verifier/The generated data processing code, execution report, and data quality verification score can be obtained from the directory.
• Adjust parameters:pass--timeoutSet the code execution timeout.--max_workersControl the number of parallel workers.--parse_reasoningOutput the model reasoning process.

DataChef project address

• GitHub repositoryhttps://github.com/yichengchen24/DataChe
• HuggingFace model libraryhttps://huggingface.co/yichengchen24/DataChef-32B
• arXiv technical paperhttps://arxiv.org/pdf/2602.11089
• Experience the demo onlinehttps://huggingface.co/spaces/yichengchen24/DataChef

DataChef's Competitive Comparison

Application scenarios of DataChef

• Domain-wide large model trainingIt automatically generates suitable data recipes for vertical fields such as mathematics, coding, finance, healthcare, and meteorology, enabling rapid construction of domain-specific models.
• Data Engineering AutomationIt replaces the traditional manual data screening and matching process that relies on expert experience, and realizes fully automated processing from raw data to training set.
• Post-training optimization of the modelGenerate high-quality fine-tuning data for existing base models to improve their performance on specific tasks.
• Data augmentation in low-resource scenariosAutomatically synthesize training samples in data-scarce domains and expand the scale of effective training data through data augmentation techniques.
• AI research automationAs the infrastructure for Automated AI Research, it supports a closed loop of data self-improvement for self-evolving AI systems.