Muse Voice Transcribe - Meta's first real-time audio perception model - AiBoss

What is Muse Voice Transcribe?

Muse Voice Transcribe is Meta's first real-time audio perception model, integrating streaming automatic speech recognition, speaker segmentation, and endpoint detection. The model supports over 70 languages and native code switching, can handle audio files exceeding one hour in length and simultaneous conversations with more than 20 people, and ranks first in both streaming transcription and speech segmentation benchmarks. Based on reinforcement learning-based adaptive latency technology, Muse Voice Transcribe dynamically balances speed and accuracy. Muse Voice Transcribe is integrated into Meta AI and Muse Code, positioned as the real-time perception layer of the AI system.

Main functions of Muse Voice Transcribe

Real-time streaming ASRThe model processes audio and outputs text in real time in 80ms increments, without waiting for the entire audio segment to finish.
Speaker segmentationAutomatically identifies and tags more than 20 different speakers, distinguishing who spoke and when.
Voice endpoint detectionIt accurately determines the start and end times of speech, supporting natural pauses and continuous dialogue.
Multilingual and code-switchingIt covers more than 70 languages and natively supports any code-switching, including mixed Chinese and English.
Long audio and multi-person conferencingIt can continuously process more than 1 hour of audio and output a complete transcription without post-processing.

The technical principle of Muse Voice Transcribe

Autoregressive multimodal architectureBased on the autoregressive model of the Muse Spark series, the audio stream is converted into soft tokens frame by frame and modeled in a unified manner with text tokens, realizing end-to-end generation from audio to text.
Streaming audio token mechanismEvery 80ms audio segment is encoded as a soft token, and the model predicts... <|next_audio|> Decide whether to continue listening to the next segment, upon receiving... <|empty_audio|> Then the remaining text is output immediately, achieving true streaming reasoning.
Task extension special tokenIntroduced based on ASR <|start_of_turn|> and <|speaker_{A-Z}|> To achieve speaker switching recognition, the following is introduced: <|speech_onset|> and <|speech_endpoint|> Endpoint detection is implemented, and all tasks share the same streaming training framework.
Reinforcement learning adaptive delayBy combining word error rate rewards and delay rewards in a multiplicative manner through RL, the model can dynamically decide whether to wait for more context or output immediately based on the recognition difficulty of each word, achieving Pareto optimality between speed and accuracy.

How to use Muse Voice Transcribe

Select entryAccess the service via three methods: Meta Model API, Meta AI for Mac, or Muse Code.
Start VoicePress in Muse Code ⌘ + v Use the keyboard shortcut, or click the voice button in the Mac version of Meta AI to start dictation.
Cross-application usePress and hold the Fn key in any Mac application to invoke Meta AI to complete the speech transcription.
API callsIntegrate into your own applications or workflows via pay-as-you-go pricing through the Meta Model API ($3 per 1000 minutes).

The core advantages of Muse Voice Transcribe

Real-time streaming ASRIt processes audio in 80ms increments, achieving true low-latency real-time transcription.
Multi-person speech splittingIt natively supports the differentiation of more than 20 speakers and can accurately assign each sentence without post-processing.
Adaptive DelayBy dynamically adjusting the waiting time for each word through reinforcement learning, Pareto optimality is achieved between speed and accuracy.
Seamless code switchingIt natively supports switching between languages within or between sentences and accurately recognizes mixed expressions such as those combining Chinese and English.
Ultra-long audio processingSupports continuous audio input of more than 1 hour, suitable for long meetings or interviews.
Voice endpoint detectionAutomatically mark the start of speech (<|speech_onset|>) and end (<|speech_endpoint|>It can accurately determine when a user has finished speaking.
Benchmark test firstIt ranked first in both the Artificial Analysis streaming speech-to-text and public speech segmentation benchmarks.

Muse Voice Transcribe project address

Project official website：https://research.meta.ai/blog/introducing-muse-voice-transcribe

Comparison of similar products to Muse Voice Transcribe

Comparison Dimensions	Muse Voice Transcribe (Meta)	Gemini 3.1 Flash Live (Google)
Product Positioning	Specialized real-time audio perception and ASR model	In general-purpose real-time multimodal dialogue models, ASR is only one of their input capabilities.
Core competencies	Streaming ASR, speaker segmentation, speech endpoint detection	Real-time two-way voice dialogue (ASR + TTS), multimodal (audio/video/image/text), function calls
Word error rate	AA-WER Streaming Index 3.1%Artificial Analysis ranked first	Specific ASR WER values have not been released; the focus is on low-latency streaming ASR and smooth dialogue.
Speaker segmentation	Native support 20+ peopleSpeaker, DER 17.5%	It does not explicitly support speaker segmentation for multiple speakers, focusing instead on single-turn/dual-turn real-time dialogue.
Delay strategy	Adaptive DelayBy using reinforcement learning, the waiting time for each word is dynamically adjusted.	Sub-200ms Fixed initial token latency for ultra-fast response.
Long audio processing	Native support 1 hour+ Continuous audio, no segmentation required	Pure audio session default 15 minutesLimitations: Session continuation mechanism required.
Code switching	Natively supports arbitrary code-switching within/between sentences, accurately recognizing mixed Chinese and English.	It supports 90+ languages, but doesn't emphasize specific optimizations for code switching.
Pure transcription cost	$0.18/hour	Audio input $0.30/hour + Text output approx. $0.07/hourApproximately $0.37/hour
Output capability	Text transcription only + speaker tagging (`<\|speaker_A\|>`）	Text + Audio Synthesis (TTS) + Tool Calls + Structured Output + Real-time Search

Application scenarios of Muse Voice Transcribe

Real-time minutes of multinational meetingsIt supports 20+ people speaking simultaneously and long meetings exceeding 1 hour, automatically distinguishing speakers and accurately transcribing discussion content that mixes Chinese and English.
AI voice assistant/AgentAs a real-time perception layer, it provides low-latency streaming ASR and accurate voice endpoint detection, allowing the AI assistant to know when the user starts and ends speaking.
Podcast and Interview Subtitle GenerationIt can process audio materials that are several hours long, natively supports multi-person dialogue segmentation, and eliminates the need for manual speaker annotation in post-production.
Multilingual customer service quality inspectionIt transcribes mixed language conversations between customer service representatives and customers in real time, improves the accuracy of technical terminology recognition by using contextual preferences, and automatically archives the data.
Developer voice input tool Use the Fn key in the Mac version of Meta AI or Muse Code to globally access speech-to-text programming, writing documents, or sending messages in any application.