Qwen2 - Alibaba Cloud's open-source next-generation general knowledge model for 1000 questions.
Qwen2 is a new generation of large language models open-sourced by the Alibaba Cloud Tongyi Qianwen team. This series covers five scale models ranging from 0.5B to 72B, adding high-quality data from 27 languages to the existing Chinese and English models, significantly improving...
What is Qwen2?
Qwen2 is a new generation of large language models open-sourced by the Alibaba Cloud Tongyi Qianwen team. This series covers five scale models ranging from 0.5B to 72B, adding high-quality data from 27 other languages to the existing Chinese and English models, significantly improving the models' natural language understanding, coding, mathematical problem-solving, and multilingual processing capabilities. Qwen2 supports context lengths of up to 128K tokens, optimizing the model's generalization and application potential. This series of models performs excellently on multiple benchmarks, surpassing Meta's Llama-3-70B, and is currently open-source on the Hugging Face and ModelScope platforms.
Qwen2 model information
| Model Name | Parameter quantity (B) | Number of non-Embedding parameters (B) | Should GQA be used? | Is Tie Embedding | Context length (tokens) |
|---|---|---|---|---|---|
| Qwen2-0.5B | 0.49 | 0.35 | yes | yes | 32K |
| Qwen2-1.5B | 1.54 | 1.31 | yes | yes | 32K |
| Qwen2-7B | 7.07 | 5.98 | yes | no | 128K |
| Qwen2-57B-A14B | 57.41 | 56.32 | yes | no | 64K |
| Qwen2-72B-Instruct | 72.71 | 70.21 | yes | no | 128K |
- ParametersThe total number of parameters in the model, expressed in billions (B).
- Non-Embedding parameter countThe number of parameters excluding the word embedding part.
- Should GQA be used?Does the model employ GQA (Generalized Query Answering) technology?
- Is Tie EmbeddingDoes the model use a technique that shares parameters between the input and output layers?
- Context lengthThe maximum context length that the model can handle, in tokens.
Qwen2 official website entrance
- Official blog introduction:https://qwenlm.github.io/zh/blog/qwen2/
- GitHub address:https://github.com/QwenLM/Qwen2
- Hugging Face address:https://huggingface.co/Qwen
- ModelScope address:https://modelscope.cn/organization/qwen
- Hugging Face online demo:https://huggingface.co/spaces/Qwen/Qwen2-72B-Instruct
Qwen2 Model Review
- The Qwen2 series performed well across multiple benchmarks, especially the Qwen2-72B model, which achieved a significant performance improvement.
- In terms of capabilities such as natural language understanding, knowledge, code, mathematics, and multilingualism, Qwen2-72B significantly outperforms current leading models, such as Llama-3-70B and Qwen1.5's 110B model.
- In 16 benchmark tests, Qwen2-72B-Instruct demonstrated a balance in basic capabilities and alignment with human values, surpassing the 72B model of Qwen1.5 and matching Llama-3-70B-Instruct.
Qwen2's model highlights
- Code & MathematicsQwen2 has significantly improved its coding and mathematical capabilities, successfully integrating the experience gained from CodeQwen 1.5 and achieving enhanced performance across multiple programming languages. Its mathematical capabilities, supported by large-scale, high-quality data, have enabled a leap forward in problem-solving abilities.
- Long text processingThe Qwen2 series of Instruct models are trained on a 32k context length and extended to longer contexts using techniques such as YARN. The Qwen2-72B-Instruct can perfectly handle information extraction tasks with a 128k context length.
- SecurityIn the category of multilingual insecure queries, Qwen2-72B-Instruct is comparable to GPT-4 in terms of security and significantly outperforms the Mistral-8x22B model, reducing the proportion of harmful responses generated.
- Multilingual abilityQwen2 performed excellently in multilingual evaluations, enhancing its ability to handle 27 languages and optimizing the language conversion problem, reducing the probability of language conversion in the model.