Skywork UniPic - Kunlun Tech's open-source multimodal unified pre-trained model

What is Skywork UniPic?

Skywork UniPic is an open-source, multimodal, unified pre-trained model from Kunlun Wanwei, possessing three core capabilities: image understanding, text-to-image generation, and image editing. Based on an autoregressive paradigm, the model integrates a MAR encoder and a SigLIP2 backbone to build a lightweight architecture, achieving high performance with only 1.5B parameters, approaching the performance of larger models. The model employs a progressive multi-task training and optimization strategy to ensure superior performance in understanding, generation, and editing tasks, supporting smooth operation on consumer-grade GPUs and providing developers with an efficient and practical multimodal solution.

The main functions of Skywork UniPic

• Image understandingThis model understands image content based on text prompts and performs tasks such as image-text matching and question answering. It accurately captures the semantic information of images, achieving a deep understanding of them.
• Text to Image GenerationBased on text prompts input by the user, the model can generate high-quality images.
• Image editingUsers provide reference images and editing instructions, and the model modifies the images according to the instructions, such as replacing elements in the image, adjusting styles, etc., supporting a variety of complex editing operations.

The technical principles of Skywork UniPic

• Autoregressive architectureThe model continues the autoregressive paradigm of GPT-4o, processing image and text data in a sequential manner to ensure efficiency in generation and understanding tasks.
• MAR encoderIn the image generation path, the MAR encoder is used as the basis for visual representation, and the image patches are generated step by step based on the mask autoregression method to achieve high-quality image generation.
• SigLIP2 backboneIn the image understanding path, the SigLIP2 backbone network is introduced, which focuses on the extraction of semantic information and improves the model's ability to understand image content.
• Progressive multitasking trainingThe model is based on a progressive multi-task training strategy, focusing first on a single task (such as text to image), and gradually introducing understanding and editing tasks after convergence. This avoids early interference between multiple tasks and ensures that the model can achieve top performance on different tasks.
• Data and reward model optimizationUsing a carefully selected pre-trained corpus of approximately 100 million words and millions of task-tuned samples, we simultaneously built the Skywork-ImgReward and Skywork-EditReward reward models to screen high-quality data and evaluate the quality of generation and editing tasks.

Skywork UniPic project address

• GitHub repositoryhttps://github.com/SkyworkAI/UniPic
• HuggingFace model libraryhttps://huggingface.co/Skywork/Skywork-UniPic-1.5B
• Technical Papers: https://github.com/SkyworkAI/UniPic/blob/main/UNIPIC.pdf

Application scenarios of Skywork UniPic

• Creative Design and Advertising ProductionAdvertising agencies can quickly generate creative images based on copywriting, designing attractive promotional posters for new products, significantly shortening the design cycle and improving work efficiency.
• Education and Online LearningOnline education platforms use this model to generate intuitive images or animations based on teaching content, helping students better understand complex knowledge points. For example, historical events can be transformed into vivid scene diagrams to enhance the fun of learning.
• Game developmentGame developers input plot descriptions, and Skywork UniPic generates game scene and character design drawings, accelerating the development process, providing creative references for art design, and enhancing the game's visual effects.
• Cultural heritage protectionMuseums restore images of cultural relics or recreate ancient scenes based on historical documents, such as recreating the prosperity of the ancient Silk Road, to help visitors understand history more intuitively and enhance the effect of cultural inheritance.
• Smart Home and Internet of ThingsThe smart home system generates corresponding scene images based on the user's voice commands, such as a cozy living room scene, providing users with intuitive scene previews and customization services, thus enhancing the user experience.