PersonaPlex - NVIDIA's full-duplex voice-to-speech AI model

What is PersonaPlex?

NVIDIA PersonaPlex is a full-duplex dialogue AI model from NVIDIA, capable of simultaneously listening and speaking, and handling interruptions, pauses, and responses in natural conversations. Users can customize the AI's persona and voice through voice and text prompts, allowing it to play various roles, from intelligent assistants to customer service representatives. The model is trained using a combination of real-world and synthetic dialogue data, demonstrating excellent conversational naturalness and task-following capabilities. PersonaPlex performs exceptionally well in various scenarios, such as education, customer service, and emergency handling, providing users with a truly natural and human-like interactive experience. The PersonaPlex-7B-v1 model is now open-source and available for developers.

Main functions of PersonaPlex

• Full-duplex dialogue capabilityPersonaPlex can listen and speak simultaneously, supports real-time interaction, and naturally handles interruptions, pauses, and responses in conversations, making the conversations smoother and more natural.
• Character and voice customizationUsers can define roles (such as smart assistant, customer service personnel, fictional characters, etc.) through text prompts and select different voice styles and tones through voice prompts to achieve highly personalized interaction.
• Natural dialogue behaviorSupports a natural conversational rhythm, including interruptions, responses (such as "uh-huh," "okay," etc.), and appropriate pauses, making the conversation feel more like human interaction.
• Task compliance abilityIt supports performing specific tasks based on text prompts, such as providing customer service, answering questions, or conducting technical discussions, maintaining a consistent role performance throughout the conversation.

PersonaPlex Technical Principles

• Full-duplex architecturePersonaPlex is based on a full-duplex model, enabling simultaneous processing of user voice input and generation of voice output, avoiding the latency of traditional cascaded systems (ASR→LLM→TTS). The model updates its internal state while the user speaks, allowing for immediate streaming responses and low-latency interaction.
• Hybrid prompt architecture:
  • Voice promptsAudio embedding captures vocal features, speaking style, and rhythm.
  • Text prompt: Describe characters, background information, and dialogue context using natural language.
  • Joint processingVoice and text prompts are processed together to generate coherent character performances.
• Transformer architectureThe Mimi speech encoder converts audio into text tokens. The Temporal and Depth Transformers handle dialogue content and actions, determining when to pause and when to interrupt. The Mimi speech decoder converts the text tokens into output speech, supporting a 24kHz sampling rate.
• Training dataThe model uses 7303 real dialogues (1217 hours) from the Fisher English corpus to generate character descriptions via GPT-OSS-120B. The model generates dialogue scripts using a language model and synthesizes speech via Chatterbox TTS, covering various scenarios and characters. Training by combining real and synthesized dialogues enables the model to learn natural speech patterns and adhere to task requirements.

PersonaPlex project address

• Project official websitehttps://research.nvidia.com/labs/adlr/personaplex/
• GitHub repositoryhttps://github.com/NVIDIA/personaplex
• HuggingFace model libraryhttps://huggingface.co/nvidia/personaplex-7b-v1

Application scenarios of PersonaPlex

• EducationAs a wise teacher, I provide clear and engaging answers and suggestions to help students better understand knowledge.
• Customer ServiceThe model can act as a bank customer service representative or a medical receptionist, handling customer issues based on text prompts and providing professional and empathetic service.
• Entertainment and socialIt simulates various fictional characters or engages in open-ended chats, providing users with a fun and personalized social experience.
• Emergency ResponseIn simulated emergency scenarios such as space missions, provide technical support and emergency response advice in a professional role.
• Medical consultationIn medical settings, it assists in recording patient information, provides professional advice, and supports the daily work of medical staff.