MeteoRA - Nanjing University launches a high-efficiency, scalable multi-tasking embedding framework
MeteoRA is a multi-task embedding framework for large language models (LLMs) developed by a research team in the Department of Computer Science and Technology at Nanjing University. It integrates multiple task-specific LoRAs (low-rank adapters) into a single base model...
What is MeteoRA?
MeteoRA, developed by a research team in the Department of Computer Science and Technology at Nanjing University, is a multi-task embedding framework for large language models (LLMs). It integrates multiple task-specific LoRAs (low-rank adapters) into a single base model, enabling efficient parameter reuse and autonomous task switching. Based on a Hybrid Expert (MoE) architecture, MeteoRA dynamically selects the most suitable LoRA adapter for the current input using a trainable gating network, without requiring explicit task instructions. MeteoRA proposes a MoE forward acceleration strategy, significantly improving inference efficiency while maintaining low memory overhead based on custom GPU kernel operators. Experiments show that MeteoRA demonstrates performance comparable to traditional fine-tuning methods on various tasks, excelling in complex tasks and supporting the resolution of multiple sub-problems within a single inference process.
MeteoRA's main functions
- Multi-task adapter integration: Embed multiple task-specific LoRAs (low-rank adapters) into a base LLM to handle multiple tasks simultaneously.
- Autonomous task selection and switchingIt enables autonomous task switching without requiring manual specification of task intent.
- Efficient ReasoningImprove the inference efficiency of the multitasking adapter while maintaining low memory overhead.
- Composite task processingSolving multiple sub-tasks in a single inference iteration, such as continuously answering questions from multiple different domains, enhances the model's flexibility and practicality.
- ScalabilityIt supports the integration of multiple LoRA adapters, making it suitable for different tasks and fields, and expanding the application scenarios of LLM.
MeteoRA's technical principles
- LoRA (Low-Rank Adaptation)A parameter-efficient fine-tuning method based on injecting low-rank matrices (A and B) into the linear layers of an LLM, updating only the matrices to adapt to the specific task without changing other parameters of the underlying model. Each LoRA adapter contains a pair of low-rank matrices used to modify the model's output.
- Hybrid Expert (MoE) ArchitectureBased on the MoE architecture, multiple LoRA adapters are treated as different "experts," and a trainable gating network dynamically selects the expert (LoRA adapter) that best suits the current input. The gating network calculates the weight of each LoRA based on the hidden state of the input and selects the adapters with the highest weights for forward propagation.
- Dynamic gating mechanismThe gating network dynamically assigns weights to each input, determining which LoRA adapters participate in the computation. Based on a top-k selection strategy, it supports the selection of multiple adapters, enabling flexible switching and combination of tasks.
- Forward acceleration strategyIntroducing custom GPU core operators based on PyTorch and Triton. These operators improve multi-task inference speed while maintaining a low memory footprint through parallelization and optimized memory access.
- Full-mode integration: Embed the LoRA adapter into all linear layers of the Transformer architecture (including the attention module and MLP module) to more comprehensively utilize knowledge from different tasks.
MeteoRA's project address
- GitHub repository:https://github.com/NJUDeepEngine/meteora
- arXiv technical paper:https://arxiv.org/pdf/2405.13053
Application scenarios of MeteoRA
- Multi-domain Q&AIt integrates knowledge from different fields, automatically switches adapters, and accurately answers various questions.
- Multilingual dialogueIt supports translation between multiple languages, enabling smooth multilingual communication.
- Composite task processingSolve complex problems involving multiple subtasks and dynamically switch adapters to complete the tasks.
- Cross-domain knowledge integrationBy combining knowledge from different fields, we can improve our ability to handle complex tasks.
- Intelligent Customer Service and AssistantDynamically switch adapters according to user needs for rapid response and improved service quality.