Antminer Ling open-sources Ling-3.0 tiny and six basic checkpoints for Flash.
Ant Financial's Ling open-source Ling-3.0-tiny and Ling-3.0-flash have a total of six basic model checkpoints, covering the pre-training, mid-training and WSM merging stages, none of which have undergone post-training.
Ant Ling team open-sourced six basic model checkpoints for Ling-3.0-tiny and Ling-3.0-flash.
These checkpoints cover the pre-trained, mid-trained, and WSM-merged phases, none of which involve post-training, and can serve as a starting point for continuous pre-training, fine-tuning, and training strategy research. The team stated that WSM replaces learning rate decay with weighted checkpoint merging, allowing researchers to validate the strategy first in the tiny version before scaling it to the flash version.
source:Ant Ling Announcement。