AB
AB
AiBoss

All Categories

project

OmniConsistency - An image style transfer model developed by the National University of Singapore

OmniConsistency is an image style transfer model developed by the National University of Singapore that addresses the consistency problem of stylized images in complex scenes. The model is trained on large-scale paired stylized data using a two-stage training process...

What is OmniConsistency?

OmniConsistency, developed by the National University of Singapore, is an image style transfer model that addresses the consistency problem of stylized images in complex scenes. The model is trained on large-scale paired stylized data and employs a two-stage training strategy to decouple style learning from consistency learning, maintaining semantic, structural, and detail consistency across multiple styles. The model supports seamless integration with LoRA modules of any style, achieving efficient and flexible stylization effects. Experiments show that OmniConsistency exhibits performance comparable to GPT-4o, while offering greater flexibility and generalization capabilities.

The main functions of OmniConsistency

  • Consistency of styleMaintain stylistic consistency in images across multiple styles to avoid style degradation.
  • Content consistencyThe goal is to preserve the semantics and details of the original image during the stylization process, ensuring the integrity of the content.
  • Style IndependenceSeamlessly integrates with any style of LoRA (Low-Rank Adaptation) module, supporting stylization tasks of various styles.
  • flexibilityIt supports flexible layout control without relying on traditional geometric constraints (such as edge graphs, sketches, and pose graphs).

OmniConsistency's technical principles

  • Two-stage training strategyThe first stage (style learning) involves independently training multiple style-specific LoRA modules, each focusing on capturing unique details of a specific style. The second stage (consistency learning) trains a consistency module on paired data, dynamically switching between different style LoRA modules to ensure that the consistency module focuses on structural and semantic consistency, avoiding the absorption of style-specific features.
  • Consistent LoRA moduleA low-rank adaptation (LoRA) module is introduced into the conditional branches to adjust only the conditional branches, avoiding interference with the stylization capabilities of the main network. A causal attention mechanism is used to ensure that conditional tokens interact internally, while the main branches (noise and text tokens) maintain clean causal modeling.
  • Conditional Token Mapping (CTM)Low-resolution conditional images guide the generation of high-resolution images, and spatial alignment is ensured based on a mapping mechanism, reducing memory and computational overhead.
  • Feature ReuseDuring the diffusion process, the intermediate features of the condition token are cached to avoid redundant calculations and improve inference efficiency.
  • Data-driven consistency learningWe constructed a high-quality pairing dataset containing 2,600 pairs of images in 22 different styles, and learned semantic and structural consistency mappings in a data-driven manner.

OmniConsistency project address

Application scenarios of OmniConsistency

  • Artistic CreationIt applies various art styles (such as animation, oil painting, sketching, etc.) to images, helping artists quickly generate stylized works.
  • Content generationIn content creation, it can quickly generate images that match a specific style, thereby enhancing the diversity and appeal of the content.
  • Advertising designGenerate consistent images for advertising and marketing materials to enhance visual appeal and brand consistency.
  • Game developmentQuickly generate stylized characters and scenes in games, improving development efficiency.
  • Virtual Reality (VR) and Augmented Reality (AR)): Generate stylized virtual environments and elements to enhance the user experience.
Back