PodAgent - A podcast generation framework jointly launched by CUHK, Microsoft, and Xiaohongshu.

What is PodAgent?

PodAgent is a podcast generation framework jointly launched by the Chinese University of Hong Kong, Microsoft, and Xiaohongshu. Based on simulating real-life talk show scenarios, it automatically generates rich and structured dialogue content using a multi-agent collaborative system (including hosts, guests, and writers). PodAgent has built a diverse voice library for accurate matching of characters and voices, ensuring naturalness and immersion in the audio. PodAgent introduces speech synthesis technology based on Large Language Models (LLM) to generate expressive and emotional speech, making podcasts more engaging. PodAgent offers comprehensive evaluation metrics to measure the quality of generated podcasts, ensuring the professionalism and diversity of the content.

PodAgent's main functions

• Generate high-quality dialogue contentIt automatically generates rich and diverse dialogue scripts covering a wide range of topics.
• Voice Role MatchingBased on the character's personality and the background of the content, the most suitable voice is dynamically matched.
• Speech Synthesis and Enhanced ExpressionAdjust the tone, rhythm, and emotion of the voice according to the mood and context of the conversation to make the podcast more vivid.
• Generate a complete podcast structureIt supports adding appropriate sound effects and background music to generate a complete podcast structure. It also supports multilingual generation to adapt to different scenarios and audience needs.
• Evaluation and optimizationIt provides comprehensive evaluation metrics to measure the quality of generated podcasts, including the richness of the conversation, the accuracy of voice matching, and the expressiveness of the voice.

PodAgent technical principles

• Multi-agent cooperative system:
  • host: Responsible for developing dialogue outlines and guiding topic discussions.
  • GuestProvide professional insights and perspectives based on the character settings.
  • screenwriterIntegrate dialogue content and optimize script coherence and diversity.
• Sound feature analysis and matchingThe process involves building a voice library, analyzing voice features (such as timbre, intonation, and emotion), and matching the most suitable voice to each character. Voice samples are extracted using open-source datasets (such as LibriTTS and AISHELL-3), and a diverse voice library is generated based on deduplication and filtering.
• LLM-guided speech synthesisThis technique uses Large Language Model (LLM)-based speech synthesis to transform text content into natural and expressive speech. The speech style predicted by LLM is used as an instruction to guide speech synthesis models (such as CosyVoice) to generate speech that matches the emotional tone of the content.
• Comprehensive evaluation indicatorsWe have introduced a set of evaluation metrics to measure the quality of generated podcasts. These metrics include lexical diversity, semantic richness, and information density of the dialogue content, as well as the accuracy of voice matching and the expressiveness of the speech. We use LLM (Linguistic Modeling) as the evaluation tool to compare and score the generated content.

PodAgent project address

• GitHub repository:https://github.com/yujxx/PodAgent
• arXiv technical paper:https://arxiv.org/pdf/2503.00455

Application scenarios of PodAgent

• Media and Content CreationQuickly generate high-quality podcasts covering topics such as news, culture, and technology, saving creation time and costs.
• Education and LearningGenerate educational podcasts, such as language learning and academic lectures, to provide a lively and engaging learning experience.
• Enterprise promotionGuang: Create brand promotion podcasts, share product stories or industry insights, and enhance brand influence.
• Self-media and personal brandingIt helps creators quickly generate podcast content, break through creative bottlenecks, and enhance the appeal of their content.
• Entertainment and CreativityGenerates fictional stories, comedy talk shows, and other entertainment podcasts, providing an immersive auditory experience.