project
CoF - DeepMind's Visual Modeling Mindset
CoF (Chain-of-Frames) is a new concept introduced by DeepMind, analogous to "Chain-of-Thought" (CoT) in language models.
What is CoF?
Chain-of-Frames (CoF) is a new concept introduced by DeepMind, analogous to Chain-of-Thought (CoT) in language models. CoF enables video models to reason in both time and space, solving complex visual tasks by generating video frame by frame. For example, the Veo 3 model uses CoF to solve maze problems, complete symmetry tasks, or perform simple visual analogical reasoning. Similar to how language models solve problems through symbolic reasoning, CoF achieves visual reasoning by generating coherent video frames, demonstrating the potential of video models in general visual understanding.
Main functions of CoF
-
Visual reasoningBy generating video frame by frame, CoF can solve problems step by step, such as finding a path in a maze, completing a symmetry task, or performing visual analogy reasoning.
-
cross-space operation: To manipulate objects in a video, such as moving, deforming, or changing the properties of objects, while maintaining the continuity of the video.
-
General Visual UnderstandingCoF helps video models understand physical rules, abstract relationships, and dynamic changes in the visual world, enabling zero-shot learning for general vision tasks.
-
Generate a coherent videoCoF ensures that the generated video is coherent in time and space, enabling the model to generate video content that conforms to logical and physical rules.
CoF technical principle
-
Generative modelsCoF relies on large-scale generative models that are trained on massive amounts of data to learn the spatiotemporal structure and dynamic changes of videos.
-
prompt driverThe model is guided to generate a video that meets the task requirements using natural language prompts and an initial image. The prompts help the model understand the task objectives, and the initial image provides the first frame of the video.
-
Frame-by-frame reasoningThe model generates video frame by frame, with each step based on the state and cues of the previous frame for reasoning. This frame-by-frame generation method is similar to chained thinking (CoT) in language models.
-
Physical and logical constraintsVideos generated using CoF must conform to physical rules and logical consistency. For example, the motion of objects must comply with physical laws, and objects in the video cannot violate constraints of the real world.
-
Optimization and feedbackThrough repeated trials and optimizations, the model can generate more accurate videos. For example, by generating multiple videos and selecting the optimal result, the success rate of the task can be improved.
CoF's project address
- Technical Papers: https://papers-pdfs.assets.alphaxiv.org/2509.20328v1.pdf
Application scenarios of CoF
- Maze SolvingCoF can generate videos that show how an object finds its way from the starting point to the end point in a maze, planning the optimal route frame by frame.
- Visual symmetry taskCoF can generate symmetrical patterns or images, and complete the drawing of symmetrical graphics by filling in the blank parts frame by frame.
- Physics simulationIt simulates physical phenomena, such as the motion, collision, and buoyancy of objects, to generate videos that conform to the laws of physics.
- Image editingUsed in image editing tasks, such as background removal, style transfer, and colorization, it completes the editing step by step by generating video frame by frame.
- Visual analogySolve visual analogy problems, such as generating missing parts to complete a visual analogy, and find the correct solution through frame-by-frame reasoning.