Khala - An open-source AI music model developed jointly by the Central Conservatory of Music and Tsinghua University.

What is Khala?

Khala is an open-source AI music model jointly developed by the Central Conservatory of Music and Tsinghua University, designed for high-fidelity song generation. The model employs a 64-layer deep acoustic token hierarchy and supports generating complete songs based on text descriptions and lyrics. It achieves top-tier open-source performance in vocal clarity, instrument separation, and lyric rhythm alignment. The project has released its code, model weights, and deployment instructions. Local operation requires 24GB of dedicated video memory, making it currently more suitable for technical users as an open-source platform.

Khala's main functions

• Text-generated songsEnter a text description to generate a complete song including vocals and accompaniment.
• Lyrics conditional generationIt supports generating melodies and vocals based on custom lyrics, achieving synchronization between lyrics and music.
• High-fidelity audio outputThe generated audio is closer to the quality of the original in terms of transients, overtones, and instrument textures.
• Vocals and accompaniment generated separatelyBy using acoustic token hierarchical modeling, the problem of human voices and instruments becoming blurred together can be reduced.
• Time alignment controlStrengthen the timing alignment between lyrics and vocal characteristics to reduce slurred words, reversed words, and rhythmic misalignment.

Khala's technical principles

• Acoustic Token Language ModelKhala is based on an acoustic token language model. The model breaks down sound into extremely fine discrete acoustic units, allowing the model to learn the organization and sequence relationships between these units.
• 64-layer deep hierarchical structureThe model employs a 64-layer acoustic token hierarchy, essentially dissecting sound at a microscopic level. This fine-grained representation makes it easier to preserve the transient and overtone details of instruments, enhancing the clarity of the overall audio texture.
• Lyrics-Audio Time Alignment MechanismDuring training and generation, Khala reinforces the time alignment constraints between lyrics and acoustic features. By pinning lyrics, beats, and vocal variations to the same timeline, she reduces lyrical misalignment and accent drift, enhancing the naturalness of the performance.

How to use Khala

• Access the project repositorySearch for the Khala project homepage on GitHub or Hugging Face to get the code and model weights.
• Preparing the hardware environmentEnsure you have a local GPU with at least 24GB of video memory (RTX 4090 recommended).
• Install dependenciesConfigure the Python environment and related dependencies according to the deployment instructions in the repository.
• Loading model weightsDownload and load the pre-trained model weights into your local environment.
• Input condition generationThe model generates a complete song by using text prompts or lyrics files as input.
• Export and Post-processingExport the generated audio for further mixing or mastering to improve the final product.

Khala's core advantages

• Open source and freeThe code and model weights are completely open and can be deployed locally, avoiding copyright and data security risks.
• Clear voiceCompared to traditional open-source models, the pronunciation is more stable, the AI sound is less pronounced, and the listening experience is closer to that of a real person singing.
• High instrument separationThe 64-layer acoustic token structure makes the transients and overtones of each instrument clearer and less likely to become muddled.
• Lyrics rhythm alignmentThe timeline alignment mechanism significantly reduces issues such as swallowed characters, reversed characters, and misaligned rhythms.
• Academic endorsementDeveloped jointly by the Central Conservatory of Music and Tsinghua University, it combines musical expertise with engineering capabilities.

Khala's project address

• GitHub repository: https://github.com/Khala-Music-AI/Khala
• HuggingFace model libraryhttps://huggingface.co/liujiafeng/Khala-MusicGeneration-v1.0
• arXiv technical paper: https://arxiv.org/pdf/2605.01790

Khala's Competitive Product Comparison

Khala's application scenarios

• Musical prototypeMusicians can use the model to quickly generate demos and verify the effect of matching melodies and lyrics.
• academic researchResearchers in music technology and audio AI fields can use it for algorithm experiments and paper reproduction.
• Independent developer integrationDevelopers can perform secondary development based on the open-source code and embed their own music creation tools or platforms.
• Copyright-sensitive scenariosEnterprises or organizations can deploy on-premises privately to avoid copyright disputes on commercial platforms.
• Music EducationIt can be used by teachers and students in colleges and universities for teaching demonstrations and research on the principles of AI-assisted composition and acoustic modeling.