KuaiFormer - A search framework launched by Kuaishou, based on Transformer.
KuaiFormer is a Transformer-based retrieval framework developed by the Kuaishou technology team for use in large-scale content recommendation systems. It redefines the retrieval process, shifting from traditional score estimation tasks to a Transformer-driven approach...
What is KuaiFormer?
KuaiFormer is a Transformer-based retrieval framework developed by the Kuaishou technical team for use in large-scale content recommendation systems. By redefining the retrieval process, it shifts from the traditional score estimation task to a Transformer-driven "next action prediction" paradigm, effectively capturing real-time interests and extracting multiple interests, significantly improving retrieval performance. KuaiFormer uses multi-interest query tokens and an adaptive sequence compression mechanism to achieve stable training on candidate sets of hundreds of millions. It was integrated into the Kuaishou App's short video recommendation system in May 2024, serving over 400 million daily active users and significantly increasing average daily user time spent.
KuaiFormer's main functions
- Multi-interest extractionBy introducing multiple query tokens to capture users' diverse interests, the model can better understand and predict users' complex interests.
- Adaptive sequence compressionTo improve the efficiency of long sequence modeling, KuaiFormer designs an adaptive sequence compression mechanism that reduces the length of the input sequence by compressing the video sequence viewed earlier, while retaining the latest video information.
- Stable training techniquesKuaiFormer uses a custom softmax learning objective and LogQ correction method to train the model stably and maintain performance when faced with a candidate set of hundreds of millions.
- Real-time recommendationsIt responds to user requests in real time and quickly selects candidate items that are relevant to the user's real-time interests from billions of options.
KuaiFormer's technical principles
- Transformer architectureIt captures complex dependencies in user behavior sequences based on a self-attention mechanism.
- Next Action PredictionKuaiFormer transforms the search process into predicting the user's next action, helping to more accurately capture the user's real-time interests.
- Multi-interest Query TokenInspired by the [CLS] Token in BERT, we introduce multiple learnable Query Tokens and combine them with a multi-interest training strategy to extract different user interest representations from historical item sequences.
- Adaptive Item Compression MechanismKuaiFormer groups and compresses the sequence of earlier projects to reduce the length of the input sequence, while performing fine-grained modeling of the latest projects.
- Smooth In-Batch Softmax LossKuaiFormer uses In-Batch Softmax as the learning target and applies the LogQ correction method to correct sampling bias.
KuaiFormer project address
- arXiv technical paper:https://arxiv.org/pdf/2411.10057
Application scenarios of KuaiFormer
- Short video recommendationsKuaishou App's short video recommendation system analyzes users' historical viewing behavior and interests to recommend new videos that users may be interested in.
- Content discoveryIn terms of content discovery, it helps users explore and discover new content, increasing the diversity and novelty of content and improving the user experience.
- Personalized recommendationsCapture users' multidimensional interests to provide more personalized recommendations and meet their individual needs.
- Real-time recommendation systemIt responds to changes in user behavior in real time and quickly updates the recommendation list, making it suitable for scenarios that require real-time recommendation updates.
- Large-scale data processingWhen dealing with video libraries containing billions of videos, KuaiFormer's high efficiency and scalability make it an ideal choice for large-scale data processing.