FIPO - A reinforcement learning algorithm launched by Alibaba Tongyi

What is FIPO?

FIPO (Future-KL Influenced Policy Optimization) is a reinforcement learning algorithm developed by Alibaba Tongyi Labs. It quantifies the influence of each token on subsequent inference trajectories through the Future-KL mechanism, achieving token-level accurate credit allocation and solving the inference bottleneck caused by the "reward-penalty sharing" in traditional RL. Under pure RL training on a 32B model, it increases the inference length from 4000 to over 10000 tokens, and improves AIME 2024 accuracy from 50% to 58%, surpassing o1-mini and overcoming the length plateau problem in deep inference.

Main functions of FIPO

• Token-level precise credit allocationBreaking away from the traditional RL "unified settlement at the end point" model, it identifies and rewards approximately 2% of the key decision tokens in the inference chain through the Future-KL mechanism, achieving fine-grained optimization.
• Inference length can be extended autonomouslyBreaking the bottleneck of "length stagnation" (~4000 tokens) in traditional methods, it drives the model to generate deep thinking chains of 10000+ tokens, and the length is positively correlated with the accuracy.
• Intelligent Attribution and Error CorrectionQuantify the causal impact of each token on subsequent trajectories: positively reinforce "stable anchor points", negatively suppress "misleading nodes", and reduce Oops Moments where the model "self-reverses the correct answer".
• Training stability guaranteeBy employing a triple mechanism of extreme value filtering, soft decay window, and influence weight pruning, gradient explosion is prevented, ensuring the numerical stability of long-chain inference training.

How to use FIPO

• Download codeObtain the FIPO code from GitHub and configure the runtime environment (based on the VeRL framework, the configuration method is similar to DAPO).
• Prepare dataUse mathematical reasoning datasets with standard answer verification (such as DAPO-17K) to avoid the need to prepare long thought chain example data.
• Adjust parametersSet the soft decay window half-life to 32 steps, limit the influence weight to the range of 1 to 1.2 times, and enable extreme value filtering to ensure training stability.
• Start trainingWhen the training program is run, the system will automatically identify key tokens in the inference chain and accurately reward or punish them. During the training process, the length of the model's response will automatically expand from 4,000 words to more than 10,000 words.
• Model InferenceOnce the model has been trained, it can be used. After inputting a mathematical problem, the model will generate a long chain of thought processes with deep self-verification and provide the answer.

Key information and usage requirements of FIPO

• Accurate identificationTargeted rewards and penalties will be applied to only 2% of the key decision tokens in the location inference chain.
• Length BreakthroughUnder pure RL training, the inference length increased from 4000 to 10000+ tokens, and the AIME 2024 accuracy increased from 50% to 58% (surpassing o1-mini).
• No cold start required: Directly use the Base model + mathematical data for training, without the need for long thought chain examples.
• Training frameworkVeRL + Ray distributed training (32B model requires a multi-card A100/H100 cluster).
• Reasoning EnvironmentThe HuggingFace format model can be loaded in a standard PyTorch environment.

FIPO's core advantages

• Token-level precise rewards and punishmentsBy using the Future-KL mechanism to identify only 2% of the key decision tokens in the inference chain, fine-grained credit allocation is achieved, thus eliminating the reward and penalty sharing problem of the traditional RL's "unified settlement at the end point".
• Breaking through the length bottleneckBreaking the traditional "glass ceiling" of 4,000 tokens, it drives the model to autonomously generate a deep inference chain of 10,000+ tokens, with a strong positive correlation between length and accuracy.
• Pure RL zero cold startNo long thought chain example data is needed; deep reasoning ability can be stimulated directly by using the Base model and verifiable reward data for training, which greatly reduces the cost of data preparation.
• SOTA performanceIn a pure RL setting at 32B scale, AIME 2024 accuracy reached 58% (peak), surpassing o1-mini (56%) and DeepSeek-R1-Zero-32B (47%).
• Training is stable and controllableThe triple protection mechanism (extreme value filtering + soft decay window + weight pruning) effectively suppresses Future-KL variance and prevents gradient explosion and catastrophic collapse in long chain training.

FIPO project address

• GitHub repositoryhttps://github.com/qwenpilot/FIPO
• arXiv technical paper: https://arxiv.org/pdf/2603.19835

Comparison of FIPO's similar products

FIPO Application Scenarios

• Mathematics Competitions and Research SupportSolve highly difficult math problems at the AIME/IMO level, generate complete derivations with self-verification, and assist mathematicians in exploring complex proof ideas.
• Complex code generation and debugging: Handle algorithm problems that require multi-step logical decomposition (such as LeetCode difficult problems), analyze the root cause of code errors through long chain reasoning and provide a fix.
• Automatic theorem provingIn the fields of formal mathematics and logic, it constructs a rigorous chain of arguments from premises to conclusions, automatically verifying the completeness of the proof process.
• Scientific research reasoningIt assists in the derivation of complex formulas and the analysis of experimental design in fields such as physics and chemistry, and integrates multi-dimensional constraints through in-depth thinking.
• Strategic Decision-Making and Business AnalysisIt addresses business decision-making issues that require weighing multiple factors (such as investment analysis and supply chain optimization), generating decision recommendations that include risk verification.