NoteLLM - A multimodal large model framework for note recommendation launched by Xiaohongshu (Little Red Book).

What is NoteLLM?

NoteLLM is a multimodal large-scale language model framework for note recommendation launched by Xiaohongshu. Based on compressed embeddings of generated notes and automatic tag category generation, NoteLLM leverages the powerful semantic understanding capabilities of large-scale language models (LLMs), combined with contrastive learning and instruction fine-tuning techniques, to improve the accuracy and relevance of note recommendations. NoteLLM-2 builds upon NoteLLM by introducing multimodal input and, based on an end-to-end fine-tuning strategy, combines a visual encoder and LLM to address the problem of neglected visual information. NoteLLM-2 proposes two mechanisms—multimodal context learning (mICL) and late fusion—to further enhance multimodal representation capabilities and significantly improve the performance of multimodal recommendation tasks. The framework demonstrates powerful recommendation capabilities on the Xiaohongshu platform and has been applied in practical recommendation systems.

NoteLLM's main functions

• Automatically generate tag categoriesGenerate tags and categories for notes to enhance the quality of note embedding.
• Improve user experienceBased on more accurate recommendations, we can improve user engagement and satisfaction on the platform.
• Multimodal Notes RecommendationBy combining text and image information, a more comprehensive note representation can be generated, improving the accuracy and relevance of multimodal recommendations.
• Solving the problem of visual information neglectBased on multimodal context learning (mICL) and late fusion mechanisms, the ability to represent visual information is enhanced.

NoteLLM's technical principles

• Note Compression PromptDesign specific prompt templates to compress note content into a special tag, while generating labels and categories.
• Contrastive LearningBased on the co-occurrence mechanism in user behavior data, we construct relevant note pairs, compare and learn to train the model, and enhance the semantic representation of note embedding.
• Instruction TuningBased on instruction fine-tuning, LLM can better understand task requirements and generate high-quality labels and categories.
• Multimodal Context Learning (mICL)The multimodal content is separated into visual and textual parts, which are then compressed into two modal compressed words. Attention between modalities is balanced based on contrastive learning.
• Late fusionVisual information is directly fused at the output stage of LLM, preserving more original visual information and avoiding the loss of visual information caused by early fusion.
• End-to-end fine-tuningCombine any existing LLM and visual encoder, and customize an efficient multimodal representation model based on end-to-end fine-tuning, without the need for pre-training alignment.

NoteLLM project address

• GitHub repository:https://github.com/Applied-Machine-Learning-Lab/NoteLLM
• arXiv technical paper:
  • NoteLLM:https://arxiv.org/pdf/2403.01744
  • NoteLLM2:https://arxiv.org/pdf/2405.16789

Application scenarios of NoteLLM

• Personalized note recommendationsBased on user interests and behaviors, it accurately recommends relevant content from a massive amount of notes, improving the user discovery experience.
• Cold Start Notes Recommendation: Help newly published notes gain exposure quickly by recommending content based on similarity.
• Tag and category generationAutomatically generates tags and categories related to the note content, improving content searchability and helping users find content of interest more quickly.
• Multimodal content recommendationIt processes text and image information to generate more comprehensive note representations, improving the accuracy and relevance of multimodal recommendations.
• Content creation assistanceIt provides creators with creative inspiration and suggestions, such as keywords, tags, and related notes recommendations, to assist in content creation.