ConsiStory - A text-based graph method for achieving topic consistency without training

What is ConsiStory?

ConsiStory, developed by researchers at NVIDIA and Tel Aviv University, is a training-free method for generating images from text. It allows images to quickly and naturally scale to different scenes while maintaining style and subject matter, following various textual cues. The core idea of ConsiStory is to achieve subject consistency during image generation by sharing the internal activations of a pre-trained text-to-image model. This method requires no optimization or pre-training of the model, greatly simplifying the process of generating consistent images.

ConsiStory's official website entrance

• Official project homepage:https://consistory-paper.github.io/
• Arxiv research paper:https://arxiv.org/abs/2402.03286
• GitHub source code repository: Coming soon

Key features of ConsiStory

• No training requiredConsiStory does not require any optimization or customization of the pre-trained text-to-image (T2I) model, meaning users can directly use existing models to generate consistent images, greatly saving time and resources.
• Consistent topic generationThis method can generate a series of images that maintain the same thematic identity, such as the same person, animal, or object, under different text prompts. This is very useful for applications that require consistent visual elements, such as storybooks, character design, and virtual asset creation.
• Cross-frame consistencyConsiStory ensures that generated images maintain consistency in subject features, even in different backgrounds and contexts, through internal activation sharing and attention mechanisms.
• Diversity of layoutTo increase the diversity of generated images, ConsiStory employs techniques such as attention dropping and query feature blending to avoid excessive uniformity in image layout.
• compatibilityThis method is compatible with existing image editing tools such as ControlNet and can be used in conjunction with them to achieve more complex image control.
• Quick generationBecause it does not require a training step, ConsiStory can generate images quickly, about 20 times faster than the current state-of-the-art technology (SoTA).

The Technical Principles of ConsiStory

• Theme positioningAt each step of the generation process, ConsiStory first locates the subject in each generated image. This is done by analyzing the model's cross-attention features, which help identify regions in the image that may contain the subject.
• Topic-driven shared attentionConsiStory extends the self-attention mechanism, allowing queries within an image to focus not only on features of its own image but also on subject-related features in other images. This allows different instances of the same subject to influence each other during generation, maintaining consistency. To constrain background and layout consistency, ConsiStory uses a subject mask to ensure that only subject-related features are shared.
• Enhanced Layout DiversityTo maintain the diversity of generated images, ConsiStory employs two strategies: first, it mixes features from the non-consistent sampling step with the generated features; second, it introduces random attention dropout during the shared attention process to reduce excessive consistency between different images.
• Feature InjectionTo further enhance subject consistency, particularly in detail, ConsiStory introduces a feature injection mechanism. By constructing a dense correspondence graph across images (using DIFT features), ConsiStory is able to precisely align and blend features across different images to enhance subject consistency.
• Anchored images and reusable themesTo improve computational efficiency, ConsiStory can select a subset of generated images as "anchor images." During the shared attention step, only the anchor images share and receive features from the other images. This not only reduces the computational burden but also improves generation quality and allows the same subject to be reused in new scenes.
• Multi-topic consistency generationConsiStory can handle images containing multiple subjects. By simply taking the union of all subject masks, consistency across multiple subjects can be maintained within a single image.