AnyStory - A high-fidelity personalized text-to-image generation framework launched by Alitongyi

What is AnyStory?

AnyStory is an innovative text-to-image generation framework developed by Alibaba Tongyi Lab, enabling high-fidelity personalized image generation for single and multiple subjects. It models subject personalization using an "encoding-routing" approach. In the encoding stage, AnyStory combines the powerful ReferenceNet and CLIP visual encoders to encode subject features with high fidelity, capturing rich details and semantic information. ReferenceNet supports high-resolution input and aligns with the feature space of the denoising U-Net, providing a solid foundation of detail for the generated image; the CLIP visual encoder is responsible for extracting a coarse concept of the subject, ensuring that the generated image closely aligns with the text description. In the routing stage, the decoupled instance-aware subject router accurately perceives and predicts the subject's position in the latent space, guiding the injection of subject conditions and effectively avoiding the subject mixing problem common in multi-subject generation, allowing each subject to retain its unique features and details in the generated image.

Main functions of AnyStory

• High-fidelity single-body personalizationAnyStory can generate high-fidelity images with specific subjects, capturing rich details and semantic information, and closely aligning the generated images with the text descriptions.
• Multi-subject personalizationIn scenarios involving multiple subjects, AnyStory can accurately perceive and predict the position of each subject in the latent space, guide the injection of subject conditions, avoid the subject mixing problem common in multi-subject generation, and enable each subject to retain its unique features and details in the generated image.
• Text description alignmentAnyStory ensures that the generated images are closely aligned with the text descriptions by extracting a rough concept of the subject through the CLIP visual encoder, thus ensuring consistency between the generated images and the text descriptions.
• Details preservedBased on ReferenceNet, AnyStory can preserve the fine details of the subject and provide high-quality image generation support.
• Flexible subject condition injectionBy using a decoupled instance-aware subject router, AnyStory can flexibly inject subject conditions into the latent space, enabling precise control over the generated image.

AnyStory's technical principles

• Encoding stage
  • ReferenceNetReferenceNet is a general-purpose and powerful image encoder that supports high-resolution input and can extract fine details of the subject. ReferenceNet's feature space is aligned with the denoising U-Net, allowing U-Net to directly extract subject features at different depths and scales. ReferenceNet encodes the reference image using a variational autoencoder (VAE) and then extracts its features through a network with the same architecture as U-Net.
  • CLIP visual encoderThe CLIP visual encoder is responsible for extracting a rough concept of the subject, ensuring that the generated image is closely aligned with the text description. CLIP features are primarily semantic, and although the resolution is low, they provide features that are well aligned with the text embedding, triggering subject generation.
• Routing phase
  • Decoupled routing mechanismThe algorithm predicts the location of the subject in the latent space through independent branches. Given a series of segmented subject images, routing features are obtained through a CLIP image encoder and an additional single-query QFormer. These features are used to model a router through an image segmentation decoder to generate a coarse routing graph, which is then refined through a masked cross-attention module to guide the injection of subject information.
  • Instance-aware routing regularization lossTo facilitate router learning and differentiation of different subject instances, an instance-aware routing regularization loss is introduced. This loss function ensures that the router accurately routes subject features to the appropriate regions, reducing the impact on irrelevant regions.
  • Injection of main information for routing guidanceFor CLIP-encoded subject representations, decoupled cross-attention is used to integrate them into U-Net, increasing the localization constraints guided by routing. For ReferenceNet-encoded subject features, the original reference attention is used, with an attention mask induced by the routing graph added to ensure that subject features are accurately injected into the target region.
• Training phase
  • Main encoder trainingThe training process includes the QFormer, ReferenceNet, and the corresponding key-value matrices in the attention blocks. ReferenceNet is initialized using pre-trained U-Net weights, and a large amount of paired data is collected to maintain consistent subject identity but show variations in background, pose, and viewpoint for training, preventing subject overfitting.
  • Router trainingA fixed-subject encoder is used to train the router. The main training data includes additional unpaired multi-person images. The router is able to effectively generalize to general subjects, thanks to the strong generalization ability of the CLIP model and its highly compressed single-token routing features.

AnyStory's project address

• Project official website:https://aigcdesigngroup.github.io/AnyStory
• arXiv technical paper:https://arxiv.org/pdf/2501.09503v1

AnyStory Application Scenarios

• Creative PaintingArtists can use AnyStory to transform their creative ideas into visual works. AnyStory provides high-quality image generation support for creating personalized portraits for specific clients, shortening the creation cycle.
• Concept DesignIn the early stages of design, AnyStory can help artists and designers quickly generate concept art, providing a reference for subsequent detailed design.
• Character concept artGame designers can use AnyStory to quickly generate concept art for multiple characters based on the game's background story and character settings.
• Comic creationIt provides comic creators with tools to quickly generate characters and scenes, accelerating the creation process and improving the visual effects of their works.
• Personalized advertisingBased on different target audiences and marketing themes, it can quickly generate images containing specific product or brand elements to improve the attractiveness and effectiveness of advertisements.