MMSkills - A multimodal skills framework jointly launched by Shanghai Jiao Tong University and Xiaohongshu.

What is MMSkills?

MMSkills is a multimodal skill framework for general visual agents jointly launched by Shanghai Jiao Tong University and Xiaohongshu. It supports expanding reusable skills from plain text steps to multimodal procedural knowledge that includes text flow, runtime state cards, and multi-view keyframes. Through the branch loading mechanism, it efficiently calls visual evidence at runtime and brings stable improvements to multiple model families such as Gemini, Qwen, Kimi, and GLM in GUI and game tasks such as OSWorld, macOSWorld, and VAB-Minecraft.

Main functions of MMSkills

• Multimodal skill pack buildingIntegrate text flow, runtime status cards, and multi-view keyframes to form reusable visual procedural knowledge units.
• Automated skill generationBased on publicly available non-test interaction trajectories, a multimodal skill library is automatically extracted through five stages: task clustering, skill planning, merging and generalization, and visual auditing.
• Branch LoadingThe main agent temporarily opens a skill branch at runtime, calling skills on demand instead of injecting the complete context all at once, thus avoiding visual evidence contaminating the main decision-making process.
• View SelectionBased on the current screenshot, historical actions, and status cards, intelligently select the most relevant keyframes and perspectives from the skill pack for alignment.
• Structured decision guidanceAfter the branch completes visual alignment, it returns compact decision support to the main agent, including applicable, subgoal, plan, do_not_do, and verify.
• Cross-domain task coverageIt supports both GUI desktop automation and visual game tasks, such as Minecraft and Super Mario Bros., enabling cross-scene transfer and reuse of multimodal skills.

The technical principles of MMSkills

• Multimodal skill pack definitionMMSkills extends traditional text skills into complete units that include the SKILL.md text flow, State_cards.json runtime state cards, and Images multi-view keyframes, enabling the Agent to determine when to use skills and how to verify progress based on visual status.
• Automated skill generationIt automatically extracts reusable diagnostic state knowledge from public trajectories through five stages: task clustering, intra-cluster skill planning, skill merging and generalization, text draft generation, visual grounding, and auditing, rather than simply saving the original presentation.
• Branch Loading Runtime MechanismWhen the main agent determines that a skill may be useful, it temporarily opens the skill branch to perform View Selection to select the key state perspective. Then, Branch Planning aligns the visual evidence with the current environment and returns compressed structured decision support.

Follow us on WeChat and reply with "open source",join inAI open source project discussion group

How to use MMSkills

• Access Skill LibraryAccess the Hugging Face dataset to download pre-generated multimodal skill packs.
• Integrated Main AgentIntegrate the MMSkills branch loading mechanism into the inference process of the existing visual agent.
• Trigger skill callThe main agent determines at runtime whether the current task may match a certain skill and initiates a branch loading request.
• Receive structured guidanceAfter the temporary branch completes the visual evidence alignment, it returns decision support such as applicable, subgoal, and plan.
• Execute and verifyThe main agent executes actions based on the returned instructions and verifies task progress through status cards.

MMSkills' core advantages

• State-aware decision makingThe skill pack contains runtime status cards, and the Agent can determine whether the current interface meets the prerequisites for the skill.
• Visual evidence alignmentMulti-view keyframes help the agent ground text conditions onto the real scene, avoiding accidental operations.
• Efficient Context ManagementBranch Loading prevents the full skill set from being directly injected into the main context, thus preventing reference images from polluting the current decision.
• The small model has significant gains.External multimodal procedural knowledge improves performance by more than double on small models such as Qwen3-VL-8B, supplementing the lack of prior knowledge in their internal processes.
• Behavioral pattern optimizationSignificantly reduces invalid clicks and repetitive actions, enabling the Agent to shift from exploratory trial and error to state-aware execution.

MMSkills project address

• Project official website: https://zkangning.github.io/MMSkills_for_Visual_Agents/
• GitHub repository: https://github.com/zkangning/MMSkills_for_Visual_Agents
• HuggingFace model libraryhttps://huggingface.co/datasets/zhangkangning/mmskills
• arXiv technical paper: https://arxiv.org/pdf/2605.13527

Comparison of MMSkills with similar products

Application scenarios of MMSkills

• Desktop GUI AutomationPerform complex cross-application operations (such as creating LibreOffice charts and modifying file permissions) in Ubuntu/macOS environments.
• Game AI agent: Perform long-distance item acquisition and crafting planning in Minecraft survival mode, or make visual-based decisions in platforming games.
• Enterprise RPA EnhancementAdd visual state perception capabilities to existing robotic process automation systems to improve the stability of cross-software operation.
• Visual task teachingThe manual operation experience is transformed into a reusable multimodal skill library for training or assisting other agents in learning.