project
MuScriptor - Kyutai and Mirelo's open-source multi-instrument music transcription model
MuScriptor is an open-source multi-instrument music transcription model jointly developed by Kyutai and Mirelo, which can automatically transcribe real-world music audio from various genres into MIDI.
What is MuScriptor?
MuScriptor is an open-source multi-instrument music transcription model jointly developed by Kyutai and Mirelo. It can automatically transcribe real-world music audio from various genres into MIDI. MuScriptor is the first model of its kind trained on a large-scale real-world dataset containing 170,000 songs. It adopts a pure decoder Transformer architecture, offers four parameter scales from 60M to 1.4B, supports instrument conditional control, and its code is open-source under the MIT license.
Main functions of MuScriptor
- Multi-instrument audio transcriptionIt automatically converts real-world music audio containing multiple instruments into standard MIDI format, supporting 36 types of instruments including piano, guitar, drums, and bass.
- Multi-genre music supportIt covers a variety of music genres, including pop, classical, rock, and heavy metal, and is suitable for complex real-world mixing scenarios.
- Instrument Condition ControlIt supports users specifying the target instrument set to be transcribed, enabling customized output and stable instrument recognition consistency across segments.
- Multi-specification model selectionIt offers four parameter scales: 60M, 103M, 307M and 1.4B, catering to both lightweight deployment and high-precision requirements.
- End-to-end automated processingUsing 5-second audio slices as input, the model automatically completes the entire process of frame segmentation, feature extraction, note detection, and MIDI generation.
Technical Principles of MuScriptor
- Pure decoder Transformer architectureThe model takes a 5-second audio slice as input, converts 16kHz mono audio into a mel-spectrogram, and predicts MIDI-like token sequences through autoregression, achieving end-to-end multi-instrument transcription.
- Three-stage training strategyThe model was first pre-trained on 1.5 million synthesized MIDI data, then fine-tuned on 170,000 real music tracks, and finally post-trained on 300 high-quality labeled data tracks through GRPO-style reinforcement learning to align with high-quality output.
- Instrument Condition ControlThe model supports specifying a target instrument set via prefix embedding, allowing users to customize the types of instruments to be transcribed, while using classifier-free guidance to stabilize instrument assignment consistency across fragments.
- Data synthesis and enhancement pipelineThe pre-training phase employs a dynamic synthesis process, performing symbol-level enhancements on MIDI such as pitch shifting, tempo changes, and instrument randomization. It synthesizes audio using over 250 soundfonts and adds random detuning to improve generalization capabilities.
Follow us on WeChat and reply with "open source",join inAI open source project discussion group
How to use MuScriptor
- Environmental preparationClone the GitHub repository and install dependencies to ensure that the system supports deep learning frameworks such as Python and PyTorch.
- Download weightObtain the model weight file with the corresponding parameter size (103M, 307M, or 1.4B) from official channels.
- Loading ModelInitialize the model using the provided inference code; you can choose whether to enable instrument condition control.
- Input audioInput the audio file to be transcribed (supports multiple formats) into the model, and the model will automatically segment it into 5-second segments for processing.
- Get MIDIThe model outputs standard MIDI files, which can be directly imported into DAW or sheet music software for further editing.
MuScriptor's core advantages
- The strongest generalization ability in the real worldCompared to models trained on synthetic data (such as MT3), MuScriptor was trained on 170,000 real multi-instrument music tracks, resulting in a significantly lower error rate in complex and realistic mixing scenarios. The official description calls it the best open-source multi-instrument transcription model to date.
- Reinforcement learning aligns high-quality outputThrough GRPO style reinforcement learning in the post-training stage, the model achieves further improvements in F1 scores at the note onset, offset, and frame levels, reducing false negatives and false positives.
- Flexible instrument condition controlUsers can specify the types of instruments to be transcribed, which can streamline the output and maintain consistency in instrument recognition across audio segments, avoiding fluctuations in instrument allocation between segments.
- Multi-specification open sourceIt offers four model sizes from 60M to 1.4B, catering to both edge device deployment and high-precision requirements. The code is MIT open source, and the weights are CC BY-NC 4.0 available.
MuScriptor's project address
- GitHub repository：https://github.com/muscriptor/muscriptor
- HuggingFace model library：https://huggingface.co/MuScriptor
- arXiv technical paper：https://arxiv.org/pdf/2607.08168v1
Comparison of MuScriptor and similar competing products
| Comparison Dimensions | MuScriptor | YourMT3+ |
|---|---|---|
| Publisher | Kyutai / Mirelo | Academic research team |
| Architecture | Pure decoder Transformer | Hierarchical Attention Transformer + MoE |
| Training data | 1.5 million synthesized MIDI tracks + 170,000 real music tracks + 300 RL tracks | Small-scale synthetic data + limited real data |
| Core Strategy | Fine-tuning with real data + training after reinforcement learning | Architecture improvements + cross-dataset enhancements |
| Instrument control | Support explicit instrument conditional control | Not supported |
| Open source license | Code MIT / Weights CC BY-NC 4.0 | Partially open source |
| Realistic audio performance | Multi F1 score of approximately 41.6 significantly outperforms the others. | As a baseline, it was surpassed. |
| Strength information | The word segmenter is not currently supported. | Depending on the specific implementation |
Application scenarios of MuScriptor
- Music production assistanceProducers can quickly convert existing audio without MIDI into editable MIDI tracks for re-arranging, mixing, or sampling.
- Automatic music score generationMusic education institutions and publishers can automatically convert recordings into sheet music, reducing the cost of manual transcription.
- Music Information RetrievalIt provides high-quality symbolic input for downstream tasks such as chord recognition, tonality analysis, and style classification.
- Generative music modelingThis provides large-scale, high-quality MIDI training data for music generation models, promoting research on symbolic music generation.
- Music archaeology and archivingIt converts unstructured audio such as historical recordings and live performances into structured MIDI data, making it easier to digitally preserve and analyze.