Sitcom-Crafter - A 3D human motion generation system jointly developed by Beihang University, the Chinese University of Hong Kong, and other universities.
Sitcom-Crafter is a 3D scene human motion generation system jointly developed by Beijing University of Aeronautics and Astronautics, the Chinese University of Hong Kong, Shenzhen, the University of Technology Sydney, Sun Yat-sen University, and other universities. It is based on user-provided long-scene guidance...
What is Sitcom-Crafter?
Sitcom-Crafter is a 3D scene human motion generation system jointly developed by Beijing University of Aeronautics and Astronautics, the Chinese University of Hong Kong, Shenzhen, the University of Technology Sydney, and Sun Yat-sen University. Based on user-provided extended storyline guidance, it generates diverse and physically realistic actions, including human walking, human-scene interaction, and human-to-human interaction. The system comprises eight modules, with three core modules responsible for motion generation and five enhancement modules for optimizing motion smoothness, synchronization, and collision correction. Sitcom-Crafter's innovative 3D scene perception technology and self-supervised SDF strategy generate physically consistent actions without additional data acquisition, significantly improving creative efficiency and providing an efficient and intelligent solution for animation and game design.
Main functions of Sitcom-Crafter
- Human walkingThe character walks naturally in the 3D scene.
- Human-scene interactionThe character interacts with objects in the environment in a way that conforms to physical logic.
- Human interaction: Coordinate action generation between characters, reduce collisions and optimize action synchronization.
- Enhanced features:
- Plot AnalysisAI automatically breaks down the script and converts it into specific character action instructions.
- Motion synchronization: Ensure that the motion generated by different modules remains consistent in time.
- Enhanced hand postureEnhance the naturalness of movement by retrieving hand gestures from a database.
- Collision CorrectionAutomatically detects and corrects collisions between characters.
- 3D redirectionThe generated motion is mapped onto the existing 3D digital character model to enhance the visual effect.
The technical principles of Sitcom-Crafter
- Motion generation module:
- Human walking generationBased on existing advanced methods (such as GAMMA), we generate walking motions of people through deep learning models.
- Human-scene interaction generationBased on methods such as DIMOS, interactive motion between characters and scene objects is generated by combining scene information.
- Human-to-human interaction generationA self-supervised scene perception method is used to synthesize SDF (signature distance function) points in a 3D scene to simulate the surrounding environment and avoid collisions between characters and the scene. Multi-person interactions are generated based on a diffusion model, taking into account the relative positions and movements between characters.
- Scene perception technology:
- SDF point synthesisBased on preprocessed motion data, define the motion region and randomly generate SDF points of surrounding objects to simulate obstacles in the scene.
- Collision detection and correctionSDF point detection detects collisions between characters and the scene or between characters and each other, and adjusts movement trajectory or speed to avoid collisions.
- Plot Comprehension and Command GenerationThe process involves parsing the plot text using a large language model (such as Gemini 1.5) and converting it into specific motion commands. Based on natural language processing technology, the complex plot is broken down into multiple motion commands and assigned to the corresponding motion generation modules.
- Enhancement Module:
- Motion synchronizationInterpolation techniques (such as Slerp) are used to ensure that the motion generated by different modules transitions smoothly over time.
- Hand gesture retrievalThe CLIP model retrieves the hand pose most similar to the text description and incorporates it into the generated motion.
- Collision CorrectionAdjust movement speed or path to avoid collisions between characters.
- 3D redirectionThe generated motion is mapped onto a high-quality 3D digital character model to enhance the visual effect.
- Unified motion representationThe system uses marker points as a unified motion representation, supports the extraction of motion data from different data sources (such as SMPL and SMPL-X models), and enhances the system's scalability and compatibility.
Sitcom-Crafter project address
- Project official website:https://windvchen.github.io/Sitcom-Crafter/
- GitHub repository:https://github.com/WindVChen/Sitcom-Crafter
- arXiv technical paper:https://arxiv.org/pdf/2410.10790
Application scenarios of Sitcom-Crafter
- Animation ProductionQuickly generate character motion, reduce manual animation workload, and support diverse action and story-driven animation design.
- Game developmentDesign natural behaviors and interactive actions for NPCs, support dynamic plot generation and real-time action feedback, and enhance game immersion.
- Virtual Reality (VR) and Augmented Reality (AR)Generate natural interactions and scene simulations for virtual characters, enhancing user experience and the naturalness of interaction.
- Film and television productionUsed in early-stage creative validation, special effects scene design, and motion capture replacement to improve production efficiency.
- Education and trainingGenerate character behaviors in simulated training, create virtual teaching assistants, and support safe demonstrations and teaching in complex scenarios.