AB
AB
AiBoss

All Categories

project

Spirit LM - Meta introduces a multimodal language model that seamlessly integrates speech and text.

Spirit LM is a multimodal language model developed by the Meta AI team that seamlessly mixes text and speech data. Spirit LM is based on a pre-trained text language model, continuously trained on text and speech units...

What is Spirit LM?

Spirit LM, developed by the Meta AI team, is a multimodal language model that seamlessly blends text and speech data. Based on a pre-trained text language model, Spirit LM extends to the speech modality through continuous training on text and speech units. The model has two versions: BASE and EXPRESSIVE. The BASE version uses speech semantic units, while the EXPRESSIVE version uses pitch and style units in addition to semantic units to simulate the expressiveness of speech. During training, Spirit LM concatenates speech and text sequences into a single tag set based on a word-level interleaving method. It can generate text with the semantic capabilities of a text model and speech with the expressive capabilities of a speech model. Spirit LM can learn new tasks across modalities, such as Automatic Speech Recognition (ASR), Text-to-Speech (TTS), and speech classification, with limited samples.

The main functions of Spirit LM

  • Cross-modal language generationSpirit LM can generate text and speech, enabling seamless switching.
  • Semantics and expressive abilityIt combines the semantic capabilities of text models with the expressive capabilities of speech models.
  • Learning from a small number of samplesIt can quickly learn new tasks such as ASR, TTS and speech classification based on a small number of samples.
  • Emotional maintenanceExpressive can understand and generate speech and text with specific emotions.
  • Multimodal understanding: Understanding and generating cross-modal content, such as converting text to speech or vice versa.

The technical principles of Spirit LM

  • Pre-training and expansionBased on a pre-trained text language model, the speech processing capabilities of the model are developed through continuous training on text and speech units.
  • Interleaved trainingThe word-level interleaving method is used to connect speech and text sequences into a single tag set for training, thereby achieving alignment between speech and text.
  • Bimodal tagging:
    • Basic version (BASE)Use speech semantic units (such as HuberT tags).
    • Expressive versionIn addition to semantic units, pitch (F0) and style units are combined to capture the expressiveness of speech.
  • Encoding and Decoding:
    • encoderConvert speech into tokens and encode speech using models such as HuBERT.
    • decoder: Restore the marker to its original modality (text or speech).
  • Dataset and Training:
    • Using large-scale text and voice datasets.
    • Interleaved training is performed based on aligned speech and text datasets.

Spirit LM project address

Application scenarios of Spirit LM

  • voice assistantIt can be integrated into smart devices to provide voice interaction functions, such as answering questions, setting reminders, and controlling smart home devices.
  • Automatic Speech Recognition (ASR)It converts speech to text, and is used in speech transcription, meeting minutes, and voice command systems.
  • Text-to-speech (TTS)It converts text information into speech output, which can be used in audiobooks, news broadcasts, and navigation systems.
  • Sentiment AnalysisAnalyze the emotional tone in voice or text for use in customer service, market research, and social media monitoring.
  • Voice translationIt enables real-time translation of voice input, facilitating cross-language communication and is suitable for international conferences and tourism scenarios.
Back