x-Algorithm - Elon Musk's open-source recommendation algorithm for the x platform.

What is x-Algorithm?

x-Algorithm is Elon Musk's open-source recommendation algorithm for the x platform, the core system for providing personalized content to the "For You" news feed. It combines posts from accounts followed by the user (In-Network) and global content retrieved through machine learning (Out-of-Network), using a Grok-based Transformer model for comprehensive ranking. The algorithm relies entirely on AI, abandoning manually designed features and rules, and predicts content a user might be interested in by analyzing their interaction history (such as likes, replies, and shares). The architecture includes Home Mixer (coordination layer), Thunder (real-time post storage and push), Phoenix (machine learning component), and Candidate Pipeline. Phoenix's dual-tower model retrieves relevant posts, while the Transformer model predicts the probability of user interaction and generates the final score. The algorithm ensures content diversity and compliance through pre-scoring filtering and post-selection processing.

The main functions of x-Algorithm

• Content integrationBy combining posts from accounts followed by the user (In-Network) and global content retrieved through machine learning (Out-of-Network), a personalized information flow is created for the user.
• AI-driven sortingThe algorithm uses a Grok-based Transformer model to predict the probability of user interaction with posts (such as likes, replies, and shares), and calculates the final score based on these probabilities to achieve accurate ranking.
• Real-time data processingThe Thunder component tracks and pushes the latest posts from accounts followed by users in real time, ensuring the timeliness of the information flow.
• Intelligent search and matchingPhoenix's dual-tower model uses user characteristics and post embeddings to perform similarity searches, quickly finding global content relevant to the user's interests.
• Filtering and OptimizationDuring the recommendation process, pre-scoring filtering and post-selection processing are used to remove duplicate, outdated, user-owned, and blocked account posts, ensuring content diversity and compliance.
• Architectural flexibilityIt adopts a composable pipeline architecture, supports parallel execution of independent stages and elegant error handling, making it easy for developers to extend and optimize the recommendation system.
• Open source sharingIt is open source under the Apache License 2.0, providing a platform for developers and researchers to learn and conduct research, and promoting the development of recommender system technology.

The technical principles of x-Algorithm

• Based on Transformer architectureUsing the Transformer architecture based on the Grok model, we can predict users' interest in content based on their interaction history (such as likes, replies, and shares).
• Dual-tower model retrievalBy embedding user features and post features, we can retrieve global content related to user interests using dot product similarity.
• Candidate isolation mechanismDuring the Transformer inference process, candidate posts can only interact with the user context to ensure consistent scores and cacheability.
• Multi-behavior predictionThe model predicts the probability of various user interactions with a post (such as liking, replying, and forwarding) and calculates the final score by combining the weights.
• Hash embedding technologyHash functions are used for embedded lookup during the retrieval and sorting stages to improve computational efficiency.
• Real-time data processing: Quickly retrieve the latest posts from accounts followed by users through memory storage and real-time push mechanisms.
• No manual feature engineeringIt relies entirely on models to learn relevance from user interaction sequences, reducing human intervention and data pipeline complexity.

x-Algorithm project address

• Github repository: https://github.com/xai-org/x-algorithm

Application scenarios of x-Algorithm

• Personalized information feed recommendationsProvide users with a precise "for you" information feed, combining user interests and global content to enhance the user experience.
• Social media content discoveryIt helps users discover new and interesting content and creators, expanding their social networks and information horizons.
• Real-time content updatesIt pushes the latest posts from accounts followed by users in real time, ensuring the timeliness and freshness of the information flow.
• Content diversity optimizationBy using an author diversity scoring mechanism, we can avoid excessive concentration of content from the same author in the information flow and provide richer information.
• Filtering of illegal contentDuring the recommendation process, illegal, spam, or content that users are not interested in is filtered out to maintain the quality of the information flow.
• Cross-platform content recommendationIt supports multiple devices and platforms, providing users with a consistent personalized recommendation experience.