ComAct - Shanghai AI Lab and others launch research paradigm for software automation. - AiBoss

What is ComAct?

ComAct (COM-as-Action) is a research paradigm that redefines the automation of professional software operations, proposed by Jiaxin Ai et al. This paradigm recognizes Windows' Component Object Model (COM) as a unified executable abstraction, transforming professional software interaction from fragile, sequential visual control to deterministic program synthesis. It addresses the error accumulation problem of GUI-based agents in long-term tasks and the limitations of API-based methods due to heterogeneous protocols and closed-source interfaces.

Main functions of ComAct

COM-as-Action Paradigm ShiftIt transforms professional software interaction from fragile sequential visual control (GUI) to deterministic program composition, utilizing the Windows COM interface as a unified executable abstraction to directly call the software's internal structured programming interface.
ComCADBench benchmarkThe study establishes the first benchmark for evaluating intelligent agents operating real industrial CAD software, comprising 1,000 tasks covering three major platforms—SolidWorks, Inventor, and AutoCAD—and seven engineering activities including 3D modeling, 2D sketching, assembly, and engineering drawings.
ComActor self-calibrating agentThe system employs a three-stage progressive training framework: text-to-code supervised fine-tuning (SFT), agent SFT based on multi-round interaction trajectories, and GRPO reinforcement learning based on continuous geometric rewards. This enables the agent to generate COM scripts, read terminal feedback, diagnose and rewrite failed steps.
ComForge Scalable Training PlatformSupports concurrent hosting of 1,000+ Dockerized Windows environments, pre-installed with real CAD software, and provides... code_execute(COM script)gui_action(Mouse/Keyboard) and reset Three action types enable large-scale parallel training and evaluation.
Cross-application workflow integrationCOM, as a unified semantic programming interface, naturally supports collaborative operation and complex workflow orchestration across various professional software (Microsoft Office, Adobe suites, industrial CAD, etc.).
Robust execution of long-range tasksBy generating a single deterministic script to replace hundreds of error-prone sequential GUI operations, it eliminates long-term error accumulation and maintains an end-to-end success rate of over 80% in long-term multi-task pipelines, significantly outperforming traditional GUI-based agents.

ComAct's technical principles

COM-as-Action Paradigm ShiftIt recognizes the Windows Component Object Model (COM) as a unified executable abstraction, redefining professional software interaction from fragile sequential visual control (GUI) to deterministic program composition, directly calling internal software functions through structured programming interfaces.
Validated instructions - code to build dataA validated instruction-COM code corpus is synthesized through a specialized data construction process, ensuring the correctness of the training data at the syntactic and semantic levels and providing a high-quality foundation for supervised fine-tuning.
Three-stage progressive training framework:
- Phase 1 (Text to Code SFT)Supervised fine-tuning is performed on a validation corpus to enable the model to learn to generate syntactically valid COM scripts;
- Phase Two (Agentic SFT)Iterative debugging and training based on multi-round interaction trajectories and screen screenshots/terminal feedback enables the agent to have the ability to recover from execution failures.
- Phase 3 (GRPO Reinforcement Learning)The method employs Group Relative Policy Optimization, combined with continuous geometric rewards based on Chamfer Distance, to directly optimize task-level geometric fidelity and bridge the gap between "grammatically correct but geometrically incompatible".
Self-correction execution loopAfter generating a COM script, the agent executes it, reads the traceback or execution result returned by the terminal, autonomously diagnoses the cause of the error and rewrites the failed steps, forming a closed loop of "generation-execution-feedback-correction".
ComForge Scalable InfrastructureA training platform built on Dockerized Windows containers, concurrently hosting over 1,000 environments pre-installed with real CAD software, supporting... code_execute(COM script)gui_action(Mouse/Keyboard) and reset Three action types enable large-scale parallel training and evaluation.
Unified Action Space Across ApplicationsLeveraging the semantic object-level nature of COM, it unifies the heterogeneous interfaces of different professional software (SolidWorks, AutoCAD, Inventor, Microsoft Office, etc.) into a standardized executable action space, naturally supporting cross-application workflow orchestration.

How to use ComAct

Environmental Preparation (ComForge)Deploy a Dockerized Windows container environment, pre-installed with target professional software (such as SolidWorks, AutoCAD, Inventor), and ComForge supports concurrent hosting of 1,000+ parallel instances for training or evaluation.
Task command inputInput natural language commands (such as "create a cube with rounded corners and generate an engineering drawing") into the ComActor agent, and the agent will parse them into the corresponding COM operation target.
COM script generationComActor, based on its three-stage training capability, automatically generates syntactically valid COM scripts (such as Python calling COM interfaces) to directly manipulate the software's internal object model, rather than simulating mouse clicks.
Execution and Feedback LoopThe script is executed in the ComForge environment. The agent reads the execution results or error traceback returned by the terminal, autonomously diagnoses the problem and rewrites the failed steps, forming a self-correcting closed loop.
Result verification and outputAfter the task is completed, the agent outputs the final generated file (such as a 3D model or engineering drawing) and verifies the geometric fidelity through the Chamfer Distance mechanism.
Cross-application workflow orchestration: Utilize the COM unified interface to connect multiple professional software programs to complete complex pipeline tasks (such as modeling in CAD → exporting → generating a BOM table in Excel).

ComAct's official website address

Project official websitehttps://knowledgexlab.github.io/ComAct/
GitHub repositoryhttps://github.com/KnowledgeXLab/ComAct
arViv technical paper: https://arxiv.org/pdf/2606.13239

ComAct's core advantages

Deterministic procedural synthesis replaces fragile visual controlThis transforms professional software interaction from sequential GUI operations that rely on pixel positioning to generating single deterministic COM scripts, fundamentally eliminating the accumulation of long-term errors and avoiding the cascading failures of traditional GUI-based agents where "one mistake leads to another".
Unified executable abstraction across heterogeneous interfacesIt leverages Windows COM as a unified semantic programming interface, integrates heterogeneous protocols from various professional software such as SolidWorks, AutoCAD, Inventor, and Microsoft Office, and naturally supports complex workflow orchestration across applications.
Solving the problem of a lack of tools for closed-source softwareCOM interfaces expose the internal functionality of applications at the semantic object level, providing high-fidelity local API-level access capabilities even to commercial software whose source code is not publicly available, thus overcoming the bottleneck of API-based methods being limited by inaccessible commercial interfaces.
Significantly superior robustness for long-range tasksMaintaining long-range multitasking pipelines End-to-end success rate of over 80%The traditional GUI-based baseline model achieves a task success rate close to that of traditional GUI-based models on ComCADBench. 0%This demonstrates exceptional resilience in long-term planning and execution.
Self-correction and iterative debugging capabilitiesTrained using Agentic SFT, the agent possesses a closed-loop capability of "generation-execution-reading traceback-diagnosis-rewriting," enabling it to autonomously recover from COM script execution failures without human intervention.
Directly optimize geometric fidelityThe third stage of GRPO reinforcement learning introduces continuous geometric rewards based on Chamfer Distance, which directly optimizes task-level geometric accuracy and bridges the execution gap of "grammatically correct but geometrically incompatible".

Application scenarios of ComAct

Industrial CAD Automated DesignIn real industrial CAD software such as SolidWorks, AutoCAD, and Inventor, natural language requirements are automatically converted into COM scripts for execution, covering the entire process of engineering activities, including 3D modeling, 2D sketching, assembly design, engineering drawing generation, quality attribute analysis, and interference detection.
Long-range multi-task pipeline orchestrationIt connects complex design pipeline tasks, such as conceptual modeling → detailed part design → assembly verification → engineering drawing output → BOM generation, and replaces hundreds of error-prone sequential GUI operations with a single deterministic script, maintaining a high end-to-end success rate.
Cross-application collaborative workflowIntegrate Microsoft Office (Excel, Word), Adobe suites, and industrial CAD software using the COM unified interface to achieve cross-application automation such as "automatically updating the Excel bill of materials and generating a Word technical report after a CAD model change".
Design verification and quality testingIt automatically performs analysis tasks such as interference detection, geometric dimension verification, and quality attribute calculation. Through a continuous geometric reward mechanism, it ensures the geometric fidelity of the output results and helps engineers quickly identify design conflicts.
Parametric Batch ModelingIt can generate or modify standardized parts/assemblies in batches based on input parameters, which is suitable for scenarios that require a large number of repetitive modeling tasks (such as standard parts library generation and series product variant design), and significantly improves design efficiency.