Apertus - Switzerland's first open-source large-scale language model

What is Apertus?

Apertus is Switzerland's first large-scale, open, multilingual language model, jointly developed by EPFL, ETH Zurich, and the Swiss National Supercomputing Center (CSCS). Available in 70B and 8B parameter versions, it is trained on a large scale using languages, with 40% of the data being non-English, including Swiss German and Romansh—languages previously underestimated in LLMs. Apertus uses a decoder-only Transformer architecture based on the novel xIELU activation function and AdEMAMix optimizer. The model is completely open; users can access the model weights, data, and training details on their own servers, maintaining control over their data.

Apertus's main functions

• Text generationGenerate coherent and relevant text based on user input prompts.
• Multilingual supportIt supports more than 1,811 languages, including many lesser-known languages that were previously underestimated in LLM.
• Transparency and opennessThe model weights, data, and training details are completely public, and users can use them on their own servers.
• Long context processingIt supports long context processing, making it suitable for complex tasks.

Apertus's technical principles

• Model ArchitectureApertus uses a dense decoder-only Transformer architecture, with models available in 8B and 70B sizes, featuring 32 layers/32 attention heads and 80 layers/64 attention heads respectively. It employs the xIELU activation function, RMSnorm normalization, RoPE positional encoding, and grouped query attention mechanism to improve model efficiency and long context handling capabilities.
• Pre-training objectivesUsing the Goldfish objective function, partial labeling is randomly masked to prevent the model from learning precise context mappings, effectively suppressing word-for-word recall while preserving downstream task performance. Pre-training data is sourced entirely from publicly available sources, respecting the opt-out intentions of content owners and avoiding the use of copyrighted, unlicensed, toxic, or personally identifiable content.
• pre-training dataApertus is pre-trained on over 15 trillion labeled datasets, covering more than 1,800 languages. The data sources are diverse, including high-quality web crawled data, code data, and mathematical data. Multiple filtering mechanisms, such as respecting crawling restrictions in the robots.txt file and removing personally identifiable information and toxic content, ensure data compliance. To improve the model's multilingual capabilities and data diversity, a significant amount of non-English content is allocated to the pre-training data.
• Training processTraining was performed using the AdEMAMix optimizer and WSD learning rate scheduling to ensure training stability and efficiency. By progressively increasing the context length, the model was able to handle longer text sequences, supporting contexts with up to 65,536 tokens.
• Post-trainingThe QRPO algorithm optimizes the model's behavior through instruction fine-tuning and alignment training, making it safer, more useful, and more aligned with human values when generating text. The post-training phase enables the model to better understand and generate text that conforms to instructions.

Apertus project address

• Project official websitehttps://www.swiss-ai.org/apertus
• HuggingFace model library: https://huggingface.co/collections/swiss-ai/apertus-llm-68b699e65415c231ace3b059
• Technical Papershttps://github.com/swiss-ai/apertus-tech-report

Application scenarios of Apertus

• Multilingual dialogue systemIt is suitable for building multilingual chatbots, customer service systems, etc., to provide users with cross-language communication and information access services.
• Code generation and assistanceIt generates code snippets based on natural language descriptions, helping developers quickly complete programming tasks and improve development efficiency. It is suitable for software development auxiliary tools.
• Education and learning supportIt generates educational content, answers academic questions, and provides learning suggestions for use in online education platforms, intelligent tutoring systems, and other educational scenarios.
• Content creationAssist in creating textual content, such as writing articles, stories, and news reports, providing inspiration and writing support for content creators.
• Translation servicesUsed in translation tasks, it provides text translation services and supports cross-language information dissemination and communication.