Miras - A general framework for deep learning architecture design launched by Google

What is Miras?

Miras is a general-purpose framework from Google for deep learning architecture design, particularly for sequence modeling tasks. Based on the concepts of associative memory and attention bias, Miras redefines models like Transformer and modern linear RNNs as associative memory modules with internal optimization objectives. Miras builds models based on four key choices: associative memory architecture, attention bias objective, retention gates, and memory learning algorithms. Miras can generate novel sequence models with different advantages, such as Moneta, Yaad, and Memora. These models perform exceptionally well in tasks like language modeling and commonsense reasoning, outperforming existing Transformer and linear RNN models.

Miras's main functions

• Unify existing architectureIt integrates various existing sequence models (such as Transformer, RetNet, Mamba, etc.) into a unified framework.
• Optimize memory managementBy introducing the concepts of attentional bias and retention gate, Miras can better balance learning new information and retaining old information, thereby optimizing the model's memory management ability.
• Design a new modelIt supports the design of novel sequence models with different attention biases and retention mechanisms, such as Moneta, Yaad, and Memora.
• Improve model performanceImprove the model's performance on long sequence tasks and maintain rapid parallel training capabilities.

Miras's technical principles

• Associative memoryAssociative memory is the mechanism that maps inputs (keys) to outputs (values). In Miras, sequence models are viewed as associative memory modules that store and retrieve information based on the learned mapping between inputs and outputs. Associative memory is the core of Miras, determining how the model stores and utilizes information from sequence data.
• attention biasAttention bias is an internal optimization objective of associative memory, used to measure how a model prioritizes certain events or stimuli. It determines how the model learns the mapping between inputs (keys and values). Different attention bias objectives (such as ℓ² regression, ℓ¹ regression, Huber loss, etc.) are chosen to adjust the model's sensitivity and robustness to data.
• Keep the doorRetention is a regularization mechanism that controls how a model retains old information while learning new information. It introduces retention regularization terms (such as ℓ² regularization, KL divergence, etc.) to balance learning and retention. This prevents the model from excessively forgetting old information and maintains better performance in long sequence tasks.
• Memory learning algorithmMemory learning algorithms are used to optimize the objective function of associative memory. Common algorithms include gradient descent and momentum gradient descent. Choosing an appropriate optimization algorithm can improve the training efficiency and convergence speed of the model.

Miras project address

• arXiv technical paper:https://arxiv.org/pdf/2504.13173

Application scenarios of Miras

• Language modelingIt is used by NLP researchers and text generation developers to efficiently process long texts and capture long-distance dependencies.
• Common sense reasoningAI researchers and intelligent assistant developers can improve their ability to understand and reason about implicit information.
• Long text processingText analysis engineers and information retrieval experts optimize the efficiency of long text processing and reduce resource consumption.
• Multimodal tasksMultimodal researchers and multimedia content analysis engineers can integrate information from multiple modalities to enhance cross-modal reasoning capabilities.