Cradle - A general-purpose, computer-controlled, multimodal AI agent framework

What is Cradle?

Cradle is a multimodal AI agent framework for General Computer Control (GCC). Developed by Kunlun Tech in collaboration with top universities and institutions such as the Beijing Academy of Artificial Intelligence, Nanyang Technological University (Singapore), and Peking University, Cradle enables AI agents to directly control the keyboard and mouse like humans without training, without relying on any internal APIs, and to achieve interaction with any open-source or closed-source software. Cradle is the first AI framework to date capable of playing various commercial games and operating various software applications, and its papers, projects, and code are all open-source.

• Information gatheringExtracting multimodal information from screen images and possible sounds for decision-making.
• Self-reflectionTo assess the success or failure of previous actions and analyze the reasons for failure in order to guide subsequent actions.
• Task InferenceBased on the current environment and historical information, infer and select the best next task.
• Skill PlanningGenerate and update skills relevant to a given task to adapt to different computer operation requirements.
• Motion planningGenerates specific operations for keyboard and mouse control, translating strategies into executable commands.

• Multimodal input processingCradle can receive and process multimodal inputs of screen images and audio, simulating human perception to understand computer interfaces and environments.
• Information extraction and understandingUse large multimodal models (such as GPT-4V) to identify visual elements in images, text information, and instructions or feedback in audio.
• Self-reflection mechanismThrough the reflection module, Cradle evaluates whether previously executed actions were successful and analyzes the reasons for failure, providing a basis for adjusting strategies.
• Task Inference and PlanningCradle determines the current priority task through the task inference module and formulates new actions in the action planning module to complete the task.
• Skill generation and updatingThe skill planning module is responsible for generating new skills or updating existing skills based on the current task. Skills exist in the form of code functions and can be instantiated and executed.
• Memory and Knowledge ManagementCradle has a long-term and short-term memory system that stores past experiences and skills, which can be retrieved and applied when needed.