LongAlign - A method from the University of Hong Kong for improving text-to-image diffusion models to handle long text alignment.

What is LongAlign?

LongAlign, developed by a research team at the University of Hong Kong, is an improved method for text-to-image (T2I) diffusion models, enhancing the alignment accuracy of long text inputs. LongAlign uses segment-level coding to segment long texts, adapting to the input limitations of coding models. It also introduces decomposition preference optimization, applying different weights to reduce overfitting and enhance alignment based on the relevant and irrelevant parts of the text in the preference model. After 20 hours of fine-tuning, LongAlign significantly improves the performance of the Stable Diffusion v1.5 model on long text alignment tasks, surpassing state-of-the-art models such as PixArt-α and Kandinsky v2.2.

The main functions of LongAlign

• Long text processingBased on a segmented coding method, it processes long text inputs and overcomes the maximum input length limitation of pre-trained coding models such as CLIP.
• Text to Image AlignmentImprove the alignment between the generated image and the input text to ensure the accuracy of the image content and the text description.
• Reduce overfittingBased on preference decomposition and reweighting strategies, LongAlign reduces overfitting during fine-tuning and improves the model's generalization ability.

The technical principles of LongAlign

• Segmented codingThis function segments long texts into multiple paragraphs (or sentences), encodes each paragraph independently, and then merges the encoded results. It supports models that can handle text exceeding the maximum input length limit.
• Preference decompositionThe scoring mechanism of the preference model is analyzed, and the preference score is decomposed into two parts: text-related part (measuring the alignment of text to image) and text-independent part (evaluating other visual aspects of the image, such as aesthetics).
• Reweighting strategyTo address the overfitting problem, LongAlign proposes a strategy that assigns different weights to text-relevant and text-irrelevant parts. This strategy is based on reducing the weights of text-irrelevant parts, enhancing the model's focus on the text content, and improving alignment.

LongAlign's project address

• GitHub repository:https://github.com/luping-liu/LongAlign
• arXiv technical paper:https://arxiv.org/pdf/2410.11817

Application scenarios of LongAlign

• Artistic CreationArtists and designers generate images that match detailed descriptions, enabling more precise visual expression in digital art creation.
• Game developmentIn game design, help create images and concept art that align with the game's background story or specific scene descriptions.
• Film and entertainment industryFilmmakers and the entertainment industry generate images that match the script or storyboard descriptions for use in pre-production or visual effects design.
• Advertising and MarketingAdvertising agencies generate images that match the advertising copy or marketing campaign description, enhancing the visual appeal of the ads.
• Education and trainingIn the field of education, generating images that match the teaching materials or course content descriptions enhances the learning experience.