AB
AB
AiBoss

All Categories

project

DiffSensei - an AI comic generation framework that can generate controllable black-and-white comic panels.

DiffSensei is a comic generation framework jointly developed by researchers from Peking University, the Shanghai AI Lab, and Nanyang Technological University. It can generate controllable black-and-white comic panels. DiffSensei integrates a diffusion-based image generator and multiple...

What is DiffSensei?

DiffSensei is a comic generation framework jointly developed by researchers from Peking University, Shanghai AI Lab, and Nanyang Technological University. It generates controllable black-and-white comic panels. DiffSensei integrates a diffusion-based image generator and a multimodal large language model (MLLM) to achieve precise control over the appearance and interactions of multiple characters in comics. The framework uses a masked cross-attention mechanism and an MLLM adapter to dynamically adjust character features, including expressions, poses, and movements, based on text prompts, generating coherent and visually appealing comic panels. DiffSensei incorporates the MangaZero dataset, supporting multi-character, multi-state comic generation tasks.

DiffSensei's main functions

  • Customized comic generationIt generates comics based on user-provided character images and text prompts, and allows users to customize the character's appearance, expressions, and actions.
  • Multi-role controlThe framework supports the generation of comics with multiple characters and scenes, and handles the interaction and layout between characters.
  • Text-compatible identity adaptationBased on MLLM, the character's characteristics are dynamically adjusted according to text prompts, so that the character's performance matches the text description.
  • Precise layout controlIt uses a masked cross-attention mechanism to precisely control the layout of characters and dialogue without the need for direct pixel transmission.
  • Dataset supportIntroducing the MangaZero dataset, a large-scale labeled dataset designed for multi-character, multi-state comic generation tasks.

DiffSensei's technical principles

  • Integrating MLLM and diffusion modelsCombine MLLM as a text-compatible identity adapter and a diffusion-based image generator to generate customized comic panels.
  • Masked Cross-Attention: Copy the key and value matrix and create independent character cross-attention layers in each cross-attention layer to achieve precise control over character layout.
  • Dialog Layout Encoding: Introducing a trainable embedding layer to represent the dialogue layout, combining dialogue embedding with noisy latent representation to encode dialogue position.
  • MLLM as a feature adapterMLLM takes source character features and panel title as input, generates target character features that are compatible with the text, and dynamically adjusts the character state.
  • Multi-role feature extractionLocal and image-level features are extracted using CLIP and an image encoder. Based on feature extractor processing, fine-grained spatial features are avoided by directly encoding them from the reference image.
  • Diffusion loss and language model lossDuring MLLM training, the Language Model Loss (LM Loss) is calculated to constrain the output format, the Mean Squared Error Loss (MSE Loss) guides the target character features based on the panel title, and the Diffusion Loss is calculated to ensure that the edited features are consistent with the image generator.

DiffSensei's project address

Application scenarios of DiffSensei

  • Comic creationArtists and cartoonists can generate comic pages, quickly converting scripts into visual narratives and improving creative efficiency.
  • Personalized content generationUsers upload character images based on their own story ideas to generate personalized comic content for personal entertainment or social media sharing.
  • Education and trainingIn the field of education, creating visual stories that match the teaching content helps students better understand and remember complex concepts.
  • Movie and game pre-madeIn film production and game design, it serves as a prefabrication tool to quickly generate storyboards or concept art, facilitating early creative and visual development.
  • Advertising and MarketingMarketers create engaging comic book ads to attract their target audience in a novel way.
Back