Step-GUI - A series of AI Agent models launched by Step-Star.

What is Step-GUI?

Step-GUI is an AI Agent series model launched by StepStar, comprising the cloud-based model Step-GUI and the edge-based model Step-GUI Edge (formerly known as Gelab-Zero). The cloud-based model excels at handling complex tasks, while the edge-based model is lightweight and can run on local devices such as mobile phones, protecting user privacy. Step-GUI utilizes a self-evolving training process and a Calibrated Step Reward System (CSRS) to transform model-generated interaction trajectories into high-quality training data, significantly reducing annotation costs and improving performance. Step-GUI has demonstrated outstanding performance in multiple benchmark tests, such as AndroidWorld and ScreenShot-Pro. Step-GUI has introduced the GUI-MCP protocol to achieve standardized model-device interaction, protecting user privacy. Its performance in real-world mobile applications is evaluated through the AndroidDaily benchmark, promoting the practicality and standardization of GUI automation tools.

Step-GUI's main functions

• Automated task executionIt allows users to operate various applications like human users, such as clicking buttons, swiping the screen, and typing text, to complete tasks ranging from simple to complex.
• Cross-platform adaptationUsing the GUI-MCP protocol, Step-GUI can run on a variety of devices (such as mobile phones, computers, and in-vehicle systems) and supports different operating systems (such as Android, iOS, Windows, and macOS).
• Privacy protectionIt adopts an edge-cloud collaborative architecture, keeping sensitive data on the local device and transmitting necessary semantic information to the cloud to ensure user privacy.
• Efficient task processingBy leveraging the collaborative efforts of a large cloud-based model and smaller edge-based models, complex tasks can be processed efficiently. The cloud-based model handles high-level inference and task decomposition, while the edge-based model is responsible for specific operation execution.
• Real-world scenario simulationIt simulates the behavior patterns of real users to complete daily high-frequency tasks such as social chatting, shopping payments, and video watching, and is suitable for a variety of practical application scenarios.

The technical principles of Step-GUI

• Multimodal large language modelStep-GUI is based on a powerful multimodal language model (such as Qwen3-VL), which can understand and generate natural language commands and translate them into specific GUI operations. The model achieves precise positioning and operation of interface elements by combining visual input (such as screenshots) and language commands.
• Self-evolving training frameworkBy calibrating the Step Reward System (CSRS), the operational trajectories generated by the model are transformed into high-quality training data. CSRS ensures the accuracy and reliability of the data through trajectory-level validation signals, while improving the model's generalization ability with detailed inference chains generated by LLM.
• End-to-cloud collaborative architectureStep-GUI adopts an edge-cloud collaborative working mode. The edge model (such as Step-GUI Edge) is responsible for executing specific operation tasks on the local device, ensuring privacy protection and low-latency response; the cloud model is responsible for handling complex logical reasoning and task decomposition, providing stronger semantic understanding and generalization capabilities.
• GUI-MCP ProtocolTo achieve standardized cross-platform interaction, Step-GUI proposes the GUI-MCP (Model Context Protocol). This protocol uses a layered architecture to abstract device operations into atomic operations and composite tasks, supporting low-level operations (such as clicks and swipes) and high-level task delegation (such as "buying coffee"), while ensuring data is processed locally and protecting user privacy.
• Reinforcement learning and data iterationStep-GUI uses reinforcement learning (such as Group Relative Policy Optimization, GRPO) for training, continuously optimizing the model's decision-making capabilities through interaction with the environment. Simultaneously, through a self-evolutionary training process, the model can continuously learn from its own generated data, improving performance.

Step-GUI project address

• GitHub repositoryhttps://github.com/stepfun-ai/gelab-zero
• Technical Papers: https://github.com/stepfun-ai/gelab-zero/blob/d1cd0c7be83e234b66dbec4c5554f5fde44dce08/report/Step-GUI_Technical_Report.pdf

Application scenarios of Step-GUI

• Personal daily task automationStep-GUI can automatically reply to social messages, manage group chats, and place orders and make payments in lifestyle service applications, helping users handle daily tasks efficiently.
• Enterprise-level automation tasksThe tool can automatically complete data entry, document editing, and email sending within office software, improving enterprise office efficiency and optimizing workflows.
• Smart Home and Internet of ThingsStep-GUI can control smart home devices, set scene modes, and automate the control of devices such as lights and air conditioners, improving the convenience of home life.
• Games and EntertainmentStep-GUI can automatically complete repetitive tasks in games or automatically play, save, and comment on videos on video platforms, enhancing the entertainment experience.
• Education and LearningStep-GUI can automatically play courses and submit assignments on online learning platforms, and create tasks and record progress in learning management tools, thus facilitating learning management.