SEED-Story - Tencent's multimodal story generation model

What is SEED-Story?

SEED-Story is a multimodal story generation model developed by Tencent in collaboration with the Hong Kong University of Science and Technology and the Chinese University of Hong Kong. Based on a multimodal large language model (MLLM), it can predict text and visual tokens, and generate style-consistent images through a visual de-tokenizer. SEED-Story introduces a multimodal attention mechanism, supporting the generation of coherent stories of up to 25 sequences. SEED-Story also provides StoryStream, a large-scale high-resolution dataset for model training and evaluation.

The main functions of SEED-Story

• Multimodal story generationSEED-Story can generate long stories with coherent narrative text and consistent imagery based on a given starting image and text.
• Multimodal attention convergence mechanismSEED-Story proposes a multimodal attention convergence mechanism to generate stories in an efficient autoregressive manner, capable of generating stories with up to 25 sequences.
• Large-scale datasets SEED-Story has released a large-scale, high-resolution dataset called StoryStream for training models and quantitatively evaluating multimodal story generation tasks from different perspectives.
• Story command adjustmentDuring the story generation process, SEED-Story fine-tunes the model through instruction adjustment, enabling it to predict the next image and the next sentence of text in the story.
• Visualization and InteractionSEED-Story also provides a visual comparison, showing that the generated images have higher quality and better consistency compared to other baseline models.

The technical principles of SEED-Story

• Multimodal Large Language Model (MLLM):Utilize large-scale language models to understand, generate, and predict text and visual tags.This model can process and generate text data, while also understanding and generating visual content.
• Text and visual tag prediction:The SEED-Story model can predict textual tags (elements or words in text) and visual tags (elements in images).
• Visual de-tokenizer:The text and visual tokens generated by the language model are converted into images, and images with consistency and style are generated through a visual de-tokenizer.
• Multimodal attention mechanism:A multimodal attention mechanism is introduced to enable the model to focus on the relationship between text and images when generating stories.This mechanism allows the model to handle the task of generating long sequences more efficiently.
• Autoregressive generation:The model uses an autoregressive approach to generate stories, meaning that each newly generated tag depends on all previously generated tags to maintain the coherence of the story.
• Long sequence generation capability:Through a multimodal attention mechanism, SEED-Story is able to generate story sequences that are longer than the training sequences.
• StoryStream dataset:We provide a large-scale, high-resolution dataset for training models and evaluating multimodal story generation tasks.The dataset contains visually compelling high-resolution images and detailed narrative text.
• Training process:This includes pre-training a de-labeler to reconstruct the image, sampling interleaved image text sequences for training, and using MLLM regression image features to adjust the generated image.

SEED-Story project address

Application scenarios of SEED-Story

• Creative Writing AssistanceIt helps writers and content creators generate story outlines, character descriptions, and plot developments, inspiring creative ideas.
• Education and LearningAs a teaching tool, it helps students understand the narrative structure of stories and improve their writing and creative thinking skills.
• Entertainment and game developmentGenerate dynamic storylines and character backgrounds in video games to provide an immersive experience.
• Advertising and MarketingQuickly generate engaging advertising stories or marketing materials to enhance the appeal and personalization of your ad content.
• Film and animation productionIt assists in script creation, generates character dialogues and scene descriptions, and accelerates the production process.