Fox-1 - TensorOpera's open-source small language model series

What is Fox-1?

Fox-1 is a series of small language models (SLMs) launched by TensorOpera, including Fox-1-1.6B and Fox-1-1.6B-Instruct-v0.1. The Fox-1 model is pre-trained on 3 trillion web-fed documents and fine-tuned on 5 billion instruction follows and multi-turn dialogue data. Fox-1 employs a 3-stage data curriculum and a deep architecture design, featuring an expanded vocabulary of 256K words and a GQA mechanism, improving efficiency and performance. The Fox-1 model demonstrates superior performance on multiple standard language model benchmarks, including the ARC Challenge, HellaSwag, MMLU, and GSM8k, outperforming models with twice the number of parameters.

Main functions of Fox-1

• Text generation and understandingFox-1 can process and generate various text tasks, such as text summarization, translation, question answering, etc.
• Instructions followedFox-1-1.6B-Instruct-v0.1 is specifically tweaked for instruction-following tasks, enabling it to understand and execute direct user commands.
• Multiple rounds of dialogueThe model is fine-tuned on multi-turn dialogue data and can be used in dialogue systems to provide coherent and relevant responses.
• Long context processingUsing Rotary Positional Embeddings (RoPE) and a 3-stage data curriculum, Fox-1 can effectively process sequences up to 8K in length, making it suitable for handling long documents and lengthy texts.
• High-efficiency reasoningFox-1 achieves inference speed and throughput comparable to larger models while maintaining a smaller model size.

The technical principles of Fox-1

• 3-stage data courseFox-1's pre-training includes a novel 3-stage data curriculum that progressively increases the block length of training samples from 2K to 8K, optimizing long context processing capabilities.
• Deep Architecture DesignFox-1 employs a deep architecture with 32 self-attention layers, which is deeper than other models and enhances the model's reasoning ability.
• Grouped Query Attention (GQA)Grouping query headers, with each group sharing the same key-value header, improves training and inference speed and reduces memory usage.
• Shared input/output embeddingFox-1 shares input and output embedding layers, reducing the number of model parameters and improving weight utilization.
• Expanding vocabularyThe model uses a vocabulary of 256K words, which, compared to the standard vocabulary, can encode information more effectively, reduce the probability of unknown words, and improve the performance of downstream tasks.
• PrenormalizationFox-1 uses RMSnorm for pre-normalization, which helps improve training efficiency.
• RoPE Location CodeFox-1 uses RoPE, which helps encode the relative positional dependencies between tokens.

Fox-1 project address

• Project official website:tensoropera-unveils-fox
• HuggingFace model library:
  • https://huggingface.co/tensoropera/Fox-1-1.6B
  • https://huggingface.co/tensoropera/Fox-1-1.6B-Instruct-v0.1
• arXiv technical paper:https://arxiv.org/pdf/2411.05281

Application scenarios of Fox-1

• Chatbots and Customer ServiceBuild chatbots to provide customer consultation services, handle multi-turn conversations, and answer user questions.
• Content creation and editingIn the content industry, we help generate creative copy, edit and polish articles, and provide writing advice.
• Language translationIt is used in the field of machine translation to help users translate between languages.
• Education and LearningIn the field of education, it serves as a teaching aid, providing support for language learning, including grammar checks and writing tutoring.
• Information retrieval and question answering systemsIt can be integrated into search engines and question-answering systems to provide fast and accurate information retrieval and answer generation.