MoonCast - A zero-sample AI podcast generation project that synthesizes a natural podcast style.

What is MoonCast?

MoonCast is a zero-shot AI podcast generation project that synthesizes natural podcast-style speech from plain text sources. Trained using a long-context language model and large-scale speech data, it can generate podcast audio several minutes long, supporting both Chinese and English. The generated speech maintains high quality in terms of naturalness and coherence, even in long audio generation. MoonCast uses specific LLM prompts to generate podcast scripts, which are then converted into the final podcast audio by its speech synthesis module. Users can quickly generate podcasts using simple commands and pre-trained weights.

MoonCast's main functions

• Long audio generationIt employs an audio modeling method based on long-context language models, which can generate podcast audios several minutes long based on large-scale long-context speech data.
• Enhance naturalnessThe podcast generation module generates scripts with natural details, which are crucial for generating natural podcast audio. Experiments show that it significantly outperforms existing baseline models in terms of naturalness and coherence.
• Multilingual supportSupports the generation of Chinese and English podcasts, and uses specific LLM hints to generate podcast scripts.
• Zero-sample speech synthesisBased on a few seconds of reference audio, it can synthesize realistic speech and maintain good speech quality and speaker similarity when processing long audio.

MoonCast's technical principles

• Multi-stage trainingThe training process of MoonCast is divided into three stages:
  • Phase 1The model learns to generate short sentences and single-person speech, mastering zero-shot speech synthesis capabilities.
  • Phase TwoThe model processes long, non-verbal audio files such as e-books, improving the stability of generating long contexts.
  • Phase ThreeThe model learns to generate long conversational audio with rich spoken details and masters sophisticated podcast generation techniques.
• Short-segment level autoregressive audio reconstructionMoonCast innovatively adopted...Short-segment level autoregressive audio reconstruction techniquesIt allows the model to stream and reconstruct the current short audio segment based on the already reconstructed content, improving the coherence of audio reconstruction.
• Spontaneous enhancementTo enhance the spontaneity of podcasts, MoonCast uses a podcast generation module to generate scripts with spontaneous details. These details include filler words, response words, and random pauses, making the conversations more natural and authentic.

MoonCast's project address

• Project official website:https://mooncastdemo.github.io/
• Github repository:https://github.com/jzq2000/MoonCast
• arXiv technical paper:https://arxiv.org/pdf/2503.14345
• Experience the demo online:https://huggingface.co/spaces/jzq11111/mooncast

Application scenarios of MoonCast

• Content creationMoonCast can transform various text content, such as stories, technical reports, news, etc., into engaging podcast audio.
• EducationIn the field of education, MoonCast can transform teaching materials (such as academic papers, e-books, etc.) into podcast format to help students better understand and absorb knowledge.
• Entertainment industryMoonCast can generate podcasts with a natural conversational style, suitable for creating entertainment content.
• Business applicationsIn the business world, MoonCast can be used to generate podcasts for internal corporate training materials, or to convert press releases, product introductions, and other materials into audio format for marketing and customer communication.
• Personal useFor individual users, MoonCast can help them turn their blogs, diaries, and other content into podcasts, making it convenient to listen to while driving, exercising, or in other scenarios.