LatentLM - A multimodal generative model jointly developed by Microsoft and Tsinghua University

What is LatentLM?

LatentLM is a multimodal generative model jointly developed by Microsoft Research and Tsinghua University. It can uniformly process discrete data (such as text) and continuous data (such as images and audio). The model uses a variational autoencoder (VAE) to encode continuous data into latent vectors, and then introduces an autoregressive vector generation technique called word diffusion. LatentLM is based on a causal Transformer architecture to achieve information sharing between different modalities, improving the model's performance and scalability in multimodal tasks. LatentLM introduces σ-VAE to solve the variance collapse problem, enhancing the robustness of autoregressive modeling, and demonstrating outstanding performance in multiple fields such as image generation, large-scale multimodal language models, and text-to-speech synthesis.

Main functions of LatentLM

• Multimodal data processingIt can process both discrete data (such as text and code) and continuous data (such as images, audio, and video).
• Unified generation and understanding interfaceIt provides an interface to unify the generation and understanding of multimodal data, such as generating any combination of text, images, audio, and video.
• Autoregressive generationBased on next-token diffusion technology, the model autoregressively generates latent vectors for continuous data.
• High-performance image generationIn image generation tasks, it is comparable to models based on diffusion or discrete labels.
• Multimodal large language model ensemble: Integrate into large multimodal language models to improve the performance of language models in multimodal tasks.
• Text-to-speech synthesisIn the field of text-to-speech synthesis, it achieves performance superior to existing state-of-the-art models with fewer decoding steps.

LatentLM Technical Principles

• Variational Autoencoder (VAE): Continuous data is encoded into latent vectors using VAEs, and the vectors are then reconstructed into the original data by a decoder.
• Next-Token Diffusion: An autoregressive method for generating latent vectors, wherein the diffusion head generates latent vectors based on each Transformer hidden state.
• Causal TransformerUse causal Transformers to process discrete and continuous data, enabling models to autoregressively predict the next element in a sequence.
• σ-VAETo address the variance collapse problem, LatentLM proposed σ-VAE, which improves the robustness of the model in autoregressive modeling by maintaining a fixed variance in the latent space.
• Mixed-modal trainingDuring training, different types of data are processed, including plain text data, image-text pair data, and interleaved image-text data.
• Efficient reasoning processDuring inference, a highly efficient decoding process is achieved through a single pass of the Transformer backbone and multiple denoising steps using a lightweight diffuser.

LatentLM's project address

• GitHub repository:https://github.com/microsoft/unilm/tree/master/LatentLM
• arXiv technical paper:https://arxiv.org/pdf/2412.08635

Application scenarios of LatentLM

• Image generationIt automatically generates corresponding images based on the text descriptions provided by the user, and is suitable for rapid prototyping in advertising design and game development.
• Intelligent Customer ServiceIn customer service, we understand users' natural language queries and provide multimodal responses that include images, text, and links.
• voice assistantIt converts users' voice commands into text and provides voice responses, suitable for smart home control and personal assistant devices.
• Automatic subtitle generationIn real time, subtitles that match the video content are generated, improving the accessibility of the content.
• Virtual streamerBased on LatentLM-generated speech and images, create virtual news anchors or virtual lecturers for instructional videos.