CLaMP 3 - A music information retrieval framework developed by the Tsinghua University team

What is CLaMP 3?

CLaMP 3 is a multimodal, multilingual music information retrieval framework developed by Professor Wenwu Zhu's team at the School of Artificial Intelligence, Tsinghua University. Based on contrastive learning, it aligns musical scores (such as ABC symbols), audio (such as MERT features), and performance signals (such as MIDI text formats) with text descriptions in multiple languages into a shared representation space. CLaMP 3 supports 27 languages and can generalize to 100 languages, making it suitable for cross-modal retrieval tasks such as text-to-music and image-to-music retrieval, zero-shot music classification, and music semantic similarity evaluation.

Main functions of CLaMP 3

• Cross-modal music retrieval:
  • Text to Music SearchRetrieves music that semantically matches a text description (supports 100 languages).
  • Image to Music RetrievalRetrieve music that matches an image based on its generated description (such as a caption generated by a BLIP model).
  • Cross-modal music retrievalSearch between different music representations (such as sheet music, MIDI, and audio). For example, search for sheet music using audio or search for audio using sheet music.
• Zero-sample music classification: No labeled data is required; music can be classified into specific categories (such as style, mood, etc.) based on semantic similarity.
• Music RecommendationsMusic recommendation is based on semantic similarity and supports recommendations within the same modality (such as audio to audio).

Technical Principles of CLaMP 3

• Multimodal data alignmentThis approach unifies music data from different modalities (such as sheet music, MIDI, and audio) and multilingual text into a shared semantic space. Based on contrastive learning, the model learns to map data from different modalities to similar vector representations, enabling cross-modal retrieval.
• Contrastive learning frameworkThe model is trained using contrastive learning (such as a variant of CLIP). The model learns to distinguish between semantically relevant and irrelevant data by using positive sample pairs (such as music and its corresponding text) and negative sample pairs (randomly paired samples), thus optimizing the representation space.
• Multilingual supportBased on XLM-R (a multilingual pre-trained model), it realizes multilingual text embedding, supports training in 27 languages, and generalizes to 100 languages.
• Training on large datasetsThe model is trained on a large-scale dataset (such as M4-RAG) containing 2.31 million high-quality music-text pairs, covering 27 languages and 194 countries.
• Feature extraction and representation:
  • Sheet musicUse Interleaved ABC notation.
  • MIDI: Convert to MIDI text format (MTF).
  • audioExtract MERT features.

CLaMP 3 project address

• Project official website:https://sanderwood.github.io/clamp3/
• GitHub repository:https://github.com/sanderwood/clamp3
• HuggingFace model library:https://huggingface.co/sander-wood/clamp3
• arXiv technical paper:https://arxiv.org/pdf/2502.10362
• Experience the demo online:https://huggingface.co/spaces/sander-wood/clamp3

Application scenarios of CLaMP 3

• Music RecommendationsBased on text descriptions or music snippets, it recommends semantically similar music and supports personalized recommendations.
• Music creation assistance:By generating matching music from text, creators can find inspiration or adjust their musical style.
• Music EducationSearch for relevant audio, sheet music, or teaching resources; supports multilingual learning.
• Music Classification and AnalysisZero-shot classification of music style, mood, etc., to assess semantic similarity of music.
• Multimedia CreationMatch suitable music to videos or images to improve content production efficiency.