Magentic-UI - Microsoft's open-source AI agent research prototype for human-computer collaboration
Magentic-UI is an open-source research prototype from Microsoft used to explore human-participatory AI agent systems. Magentic-UI is a human-centered AI agent capable of collaborating with users to complete complex web tasks, such as browsing...
What is Magentic-UI?
Magentic-UI is an open-source research prototype from Microsoft used to explore human-participatory AI agent systems. Magentic-UI is a human-centric AI agent capable of collaborating with users to complete complex web tasks, such as browsing web pages, executing code, and processing files. Its core features include co-planning, co-tasking, action guards, and plan learning. Magentic-UI provides a transparent and controllable interactive experience by allowing users to participate in task planning and execution in real time. It leverages human feedback to improve task completion efficiency, reduce manual labor costs, and provide an experimental platform for researching human-AI agent collaboration.
Main functions of Magentic-UI
- Collaborative PlanningGenerate a step-by-step plan before executing a task, which users can modify and approve to ensure the task proceeds as expected.
- Collaborative ExecutionIt displays the upcoming operations in real time, allowing users to take over control at any time and ensure that the task execution meets their needs.
- Security Mechanism: Obtain user approval before performing critical or irreversible operations, support user-defined approval policies, and ensure operational security.
- Learning and reuseAfter a task is completed, the execution plan is saved, which users can reuse or modify in future tasks to improve efficiency.
The technical principles of Magentic-UI
- System ArchitectureMagentic-UI is based on the AutoGen-based Magentic-One system, composed of multiple specialized agents that work collaboratively to complete tasks. The Orchestrator, driven by a large language model (LLM), is responsible for collaborative planning with users, deciding when to solicit user feedback, and assigning subtasks to other agents. The WebSurfer, equipped with an LLM agent that controls the browser, can perform clicks, input, scrolling, and other operations to complete the web browsing tasks assigned by the Orchestrator. The Coder, equipped with an LLM agent using a Docker code execution container, provides feedback to the Orchestrator. The FileSurfer, equipped with an LLM agent using a Docker container and file conversion tools, can locate files, convert them to Markdown format, and answer questions about the files.
- Interaction FlowThe user interacts with the Magentic-UI by inputting text messages and attaching images. The Orchestrator creates a step-by-step plan in natural language based on the user input, which the user can then modify using the plan editing interface. For each step of the plan, the Orchestrator determines which Agent or user will complete it, sends a request, and waits for a response. Once all steps are completed, the Orchestrator generates the final answer and presents it to the user. If the plan is found to be inadequate during execution, the Orchestrator, with the user's permission, re-plans and executes a new plan.
- Safety and ControlUsers can configure a list of websites that Magentic-UI can access; access to websites not on the list requires explicit user approval. Users can interrupt Magentic-UI at any stage of its task execution, stopping any pending code execution or web browsing operations. Both the browser and code executor controlled by Magentic-UI run in Docker containers to avoid impacting the host environment and prevent security risks such as login credential leakage. Users can configure action approval policies to determine whether Magentic-UI requires user approval when performing certain operations.
Magentic-UI project address
- Project official website:https://www.microsoft.com/en-us/research/blog/magentic-ui
- GitHub repository:https://github.com/microsoft/magentic-ui
Application Scenarios of Magentic-UI
- Automation of complex tasksIt helps users complete multi-step web tasks, such as comparing product prices, filling out online forms, or booking a trip.
- Code writing and execution assistanceGenerate code snippets and run code securely, for example, for data analysis or script writing.
- Document Processing and Information RetrievalConvert file formats, search file content, and answer related questions.
- Research and DevelopmentIt provides researchers with an experimental platform to explore human-machine collaboration methods.
- Education and TrainingAs a teaching tool, it helps with learning task planning and AI collaboration.