Magenta RealTime 2 - Google's open-source real-time music generation model, Magenta.

What is Magenta RealTime 2?

Magenta RealTime 2 (MRT2) is the second-generation open-source local real-time music generation model from Google's Magenta team. Based on a frame-by-frame generation architecture, MRT2 achieves an actual control latency of approximately 200 milliseconds, with each frame taking 40 milliseconds, allowing AI to react to input changes at a speed close to that of a real instrument performance. The model provides two versions: mrt2_base (2.4B parameters, high quality) and mrt2_small (230M parameters, high speed), both optimized for Apple Silicon. The smaller model can run in real-time on the M1 and subsequent chips. MRT2 employs an open-source weighting strategy and comes with free Jam and DAW plugins to lower the barrier to entry for music creators and developers.

Main features of Magenta RealTime 2

• Low-latency real-time generationAudio is generated frame by frame, with each frame taking 40 milliseconds. The actual control latency is about 200 milliseconds, which is reduced to 1/15 of the previous generation's 3-second latency, enabling true real-time interaction.
• Multimodal control inputIt supports text descriptions, audio style prompts, MIDI note input, and drum kit on/off control, allowing users to guide the music style and direction in real time through various methods.
• Continuous note followThe model can continuously follow changes in note input, adjust the generated content in real time, and adapt to the dynamic changes in the user's performance.
• Auto-Strum Intelligent PickIn Auto-Strum mode, AI automatically determines when to pluck the string or start the note; when turned off, users can precisely specify the start time of the note, balancing automation and fine-grained control.
• Drum track switchingSupports switching between output with and without drums, suitable for multi-track arrangement scenarios, and flexibly adapts to the needs of different instrument arrangements.
• Dual-scale model selectionIt offers two specifications: mrt2_base (2.4B parameters, prioritizing sound quality) and mrt2_small (230M parameters, prioritizing speed), to adapt to different hardware performance levels.
• Apple Silicon native optimization: Deeply optimized for M-series chips, mrt2_small can run in real time on M1 and later models, while mrt2_base supports M2 Max and above models.

Technical Principles of Magenta RealTime 2

• Codec Language Model ArchitectureMRT2 belongs to the codec language model paradigm. Its core idea is to discretize continuous audio signals into predictable token sequences, generating them frame-by-frame through an autoregressive approach. The model uses the SpectroStream audio codec to compress 48kHz stereo audio into a compact latent representation, outputting audio tokens at a 25Hz frame rate. This discretization strategy significantly reduces sequence length and computational complexity, enabling real-time inference on consumer-grade hardware.
• Frame-by-frame autoregressive generation mechanismUnlike its predecessor, which used a 2-second audio block batch processing design with a delay of approximately 3 seconds, MRT2 adopts a frame-by-frame generation architecture. Each frame takes only 40 milliseconds. After receiving the current input conditions (MIDI, text, audio style), the model immediately predicts the audio tag distribution of the next frame and decodes and outputs it. The streaming autoregressive approach compresses the actual control latency to approximately 200 milliseconds, a reduction to 1/15 of the previous generation, approaching the human perception threshold for musical instrument responses.
• SpectroStream audio codecSpectroStream, serving as both the audio front-end and back-end of MRT2, is responsible for encoding the raw 48kHz stereo waveform into discrete tokens that the model can process, and then decoding it back into audible audio. The codec is optimized for the high-frequency structure and stereo phase information of the music signal, ensuring that timbre, spatiality, and harmonic details are preserved even in a highly compressed potential space, providing a fundamental guarantee for sound quality in real-time scenarios.

How to use Magenta RealTime 2

• Experience the Jam appDownload the free Jam app, enter a music style description, and the model will start generating; adjust the pitch using the keyboard below, which supports mouse, MacBook keyboard, or MIDI keyboard input.
• Call in DAWInstall the MRT2 plugin to directly call the model in common digital audio workstations (DAWs), integrating AI generation into the existing music production workflow.
• Python library developmentDeveloper execution pip install magenta-rt Install the Python library to integrate MRT2 into your custom music application or interactive device via the API.
• Local edge deployment: Achieve fully offline, local inference on Apple Silicon Macs using the C++/MLX inference engine, without cloud dependencies.

Magenta RealTime 2's core advantages

• Extremely low latencyA latency of 200 milliseconds allows AI to respond at a speed close to the human perception threshold for playing musical instruments, enabling true "improvisational ensemble playing".
• Running entirely locallyOptimized based on Apple Silicon and the MLX framework, all inference is performed locally without a network connection, ensuring privacy and low-latency stability.
• Open source ecosystemThe model weights are open source, and it offers three access methods: Python library, standalone application, and DAW plugin, covering the full-chain needs from ordinary users to professional developers.
• Multi-dimensional real-time controlIt simultaneously supports text, audio, MIDI, and drum kit on/off control, providing the richest interactive control dimensions in the field of real-time generation.

Project address for Magenta RealTime 2

• Project official websitehttps://magenta.withgoogle.com/magenta-realtime-2
• GitHub repositoryhttps://github.com/magenta/magenta-realtime
• HuggingFace model libraryhttps://huggingface.co/google/magenta-realtime-2

Magenta RealTime 2 Comparison with Similar Products

Application scenarios of Magenta RealTime 2

• Live improvisationMusicians can play in real-time with the MRT2 using a MIDI keyboard. AI generates accompaniment or corresponding musical phrases in real time based on the notes played and style prompts, which can be used in music scenarios with strong improvisation, such as jazz and electronic music.
• Real-time music arrangement assistanceWhen the MRT2 plugin is loaded into DAW, the AI provides real-time feedback on the arrangement effect as the creator adjusts chord progressions or style descriptions, accelerating the creative iteration process.
• Interactive music installationDevelopers use open-source Python libraries and their low latency to build interactive sound installations in museums, exhibitions, or on stage, allowing audience movements to be converted into music in real time.
• Music Education and PracticeStudents input style descriptions through the Jam app, and AI generates accompaniment in real time for solo practice or to simulate ensemble experiences with different band arrangements.