DynaMem - A dynamic spatial semantic memory system developed by New York University and Hello Robot
DynaMem is a dynamic spatial semantic memory system developed by New York University and Hello Robot, designed specifically for mobile operations in open worlds. It maintains a feature point cloud as the robot's memory to handle dynamic changes in the environment...
What is DynaMem?
DynaMem, developed by New York University and Hello Robot, is a dynamic spatial semantic memory system designed for mobile manipulation in open worlds. It maintains a feature point cloud as the robot's memory to handle dynamic changes in the environment, such as the addition and removal of objects. When a new RGBD observation is received, DynaMem updates its memory, adding new objects and removing points that no longer exist. DynaMem locates objects based on text queries and navigates to the target object when necessary. DynaMem performs exceptionally well in handling dynamic objects, achieving a success rate of 70%, significantly higher than traditional systems.
DynaMem's main functions
- Dynamic spatial semantic memoryMaintain a dynamically updated feature point cloud as the robot's memory system to adapt to changes in the environment.
- Environmental perception and updatingWhen the robot receives a new RGBD (red, green, blue) observation, DynaMem adds the newly observed object to its memory and removes points that no longer exist.
- Text query positioningThe goal is to locate objects of interest in the environment based on a text query, by finding the point most similar to the text query and the last observed image.
- Navigation and InteractionIf the text is successfully located in the environment, navigate the robot to the target object; if the text cannot be located in the environment, explore the environment to find the target.
- Dynamic object processingDynaMem excels at handling dynamic objects in the environment, with only a very small number of experiments failing due to the inability to navigate to dynamic objects.
- Value Map ExplorationProjecting the robot's memories onto a value map guides the robot in exploring its environment.
DynaMem's technical principles
- Feature point cloud maintenanceMaintain a feature point cloud as the robot's memory; the point cloud is dynamically updated as the environment changes.
- RGBD observation fusionWhen the robot receives new RGBD observations, DynaMem integrates these observations into its existing memory to reflect the latest state of the environment.
- Text query matchingIt understands and matches text queries based on advanced visual language models (VLMs) and multimodal large language models (mLLMs), and locates objects in memory.
- Similarity assessmentThe similarity between feature points in a point cloud and a text query is used to determine the location of an object.
- Environmental NavigationIf the text query successfully locates an object in the environment, guide the robot to navigate to that object; if it fails to locate the object, use a value map to guide the robot to explore the environment.
DynaMem project address
- Project official website:dynamem.github.io
- arXiv technical paper:https://arxiv.org/pdf/2411.04999
Application scenarios of DynaMem
- Home AutomationIn a home environment, it helps service robots identify, locate, and manipulate furniture, appliances, or other household items.
- Industrial AutomationIn the manufacturing and logistics sectors, auxiliary robots are used to sort, move, and assemble items on production lines.
- Warehouse ManagementIn a warehouse environment, robots assist in inventory management, including the location, picking, and handling of goods, especially when goods are frequently moving in and out.
- Disaster reliefIn disaster relief scenarios, this helps robots navigate through ruins to find survivors or important items.
- Agricultural automationIn the agricultural sector, robots assist in the monitoring, harvesting, and handling of crops.