AB
AB
AiBoss

All Categories

project

VoiceSculptor - A timbre design model developed in collaboration with Northwestern Polytechnical University and other open-source platforms such as VoiceGraph Intelligence.

VoiceSculptor is a voice design model developed by Northwestern Polytechnical University, YTO Intelligence, and other institutions. It can achieve fine-grained control over speech synthesis through natural language commands.

What is VoiceSculptor?

VoiceSculptor is a technology developed by Northwestern Polytechnical University and the Institute of Graph Intelligence.The voice design models developed by institutions such as VoiceSculptor enable fine-grained control over speech synthesis through natural language commands. These models allow for flexible adjustments to attributes such as gender, age, speech rate, pitch, volume, and emotion, and combine retrieval-enhanced generation (RAG) technology to improve the understanding of complex commands. Audio generated by VoiceSculptor can be used for voice cloning, meeting the needs of personalized speech synthesis, virtual voices, and interactive AI applications, thus driving speech synthesis technology towards greater freedom and controllability.

VoiceSculptor's main functions

  • Natural language control timbre generationUsers can describe desired vocal characteristics, such as gender, age, speech rate, tone, and emotion, through natural language commands to achieve highly customized speech synthesis.
  • Fine-grained attribute controlThe model supports fine-tuning of various speech attributes, including gender, age, speech rate, pitch, volume, and emotional expression, to meet personalized needs.
  • Search Enhancement Generation (RAG)By using retrieval enhancement techniques, the model can better understand and generate speech corresponding to natural language instructions from outside the domain, thereby improving generalization ability and robustness.
  • Voice cloning and speech synthesisThe generated audio can be used as a cue waveform for CosyVoice2 timbre cloning and downstream speech synthesis tasks, enabling efficient timbre transfer and synthesis.
  • Role-playing and diverse voice generationIt supports generating corresponding voice styles based on different role descriptions, such as suspense novel narrators, news anchors, fairy tale narrators, etc., adapting to a variety of application scenarios.

The technical principles of VoiceSculptor

  • Overall ArchitectureVoiceSculptor consists of a Voice Design module and a Voice Clone module. The Voice Design module generates timbre and speech attributes based on the LLaSA model and uses the XCodec2 decoder to restore them to audio. The Voice Clone module uses the generated audio as a cue waveform and uses CosyVoice2 to achieve timbre cloning and speech synthesis.
  • Voice Design ModuleThe speech design module uses the LLaSA model as its foundation. By jointly training natural language commands, fine-grained attribute tokens, and target text, it maps the text into speech feature representations. Then, the XCodec2 decoder converts the feature vectors into audio waveforms, enabling natural language commands to control timbre generation.
  • Search Enhancement Generation (RAG)The model incorporates retrieval enhancement techniques, using Qwen3-Embedding-0.6B to vectorize natural language instructions and store them in the Milvus database. During inference, vector retrieval is performed on the input instructions, matching similar instructions to enhance the model's ability to understand and generate complex instructions.
  • Voice Clone ModuleThe voice cloning module is based on CosyVoice2. It takes the audio generated by the voice design module as the input prompt waveform and generates a voice similar to the prompt waveform through timbre cloning technology to complete the downstream voice synthesis task.
  • Training data and strategyThe training data includes a large number of speech samples labeled with timbre attributes. The model performance is improved by combining continuous pre-training and supervised fine-tuning to ensure the model's generalization ability and generation effect in different scenarios.

VoiceSculptor project address

  • GitHub repository: https://github.com/ASLP-lab/VoiceSculptor
  • HuggingFace model libraryhttps://huggingface.co/ASLP-lab/VoiceSculptor-VD

Application scenarios of VoiceSculptor

  • Personalized speech synthesisIt provides users with customized voice services, such as generating voices in a specific style based on user descriptions, for use in devices such as personal assistants and smart speakers, to meet users' personalized needs for voice style.
  • Virtual voices and digital humansIt generates natural and diverse voices for virtual anchors, virtual customer service representatives, and virtual characters, enhancing the expressiveness and interactivity of virtual characters and improving the user experience.
  • Audio content creationIn fields such as audiobooks, radio dramas, and animation dubbing, it can quickly generate different styles of voice based on text content, improving content creation efficiency and reducing production costs.
  • Interactive AIIt provides natural language controlled voice output for interactive AI systems such as chatbots and intelligent customer service, enhancing the naturalness and user-friendliness of the system.
  • Education and TrainingIt generates vivid audio narration for educational software, such as simulating dialogues between different characters or speeches by historical figures, thereby enhancing the fun and immersion of learning.
Back