Poetry2Image - An iterative correction framework designed specifically for generating images of classical Chinese poems.

What is Poetry2Image?

Poetry2Image, proposed by Harbin Institute of Technology, is an iterative correction framework specifically designed for generating images of classical Chinese poetry. Through automated feedback and correction loops, the framework enhances the consistency between the poem and the image, effectively capturing the semantic and artistic essence of the poem. It addresses the common problems of missing key elements or semantic confusion in text-to-image generation models when processing classical Chinese poetry. Poetry2Image improves the element completeness and semantic consistency of generated images; when combined with five popular image generation models, it achieves an average element completeness of 70.63% and semantic consistency of 80.09%.

The main functions of Poetry2Image

• Automated feedback and correctionPoetry2Image uses an external poetry dataset to create an automated feedback and correction loop that enhances the alignment between the poems and the images, improving the quality and accuracy of the generated images.
• Reduce human interventionThis framework reduces the need for extensive human intervention and expertise required in traditional methods, making the image generation process more efficient.
• Improve efficiency and qualityCompared to traditional fine-tuning methods, Poetry2Image significantly reduces training costs and time while maintaining the quality of generated images.
• Search and translationThe system will search the poetry database for ancient poems provided by the user and find their modern Chinese translations and analyses.
• Generate initial imageBased on the modern Chinese translation of the poem, the system will generate a preliminary image.
• Extract key elementsThe system will use a large language model to identify and extract key elements from the poem.
• Image CorrectionThe system will check whether the generated image contains all these key elements and make modification suggestions if necessary, such as adding missing elements or adjusting the position of elements.
• Iterative optimizationIf the elements in the image are incomplete or incorrect, the system will regenerate the image based on suggestions. This process will be repeated until the image accurately reflects the mood of the poem.

The technical principles of Poetry2Image

• Image element recognition and correctionThe system identifies element information in an image based on an Open Vocabulary Detector (OVD) and then provides modification suggestions through an LLM. These suggestions are presented as bounding boxes in the image to guide the image editing model in editing the initial image.
• Compatibility and training costPoetry2Image has no restrictions on the text-to-image generation model used for initial image generation, and the iterative correction operation eliminates the need for additional training costs, while the automated image generation and feedback process significantly reduces manual annotation.

Poetry2Image project address

• Github repository:https://github.com/prajwalppv/Poetry2Image
• arXiv technical paper:https://arxiv.org/pdf/2407.06196v1

Application scenarios of Poetry2Image

• Dissemination of Ancient Poetry CulturePoetry2Image promotes the dissemination of classical Chinese poetry culture by generating images that match the artistic conception of classical poems, enabling modern people to understand and appreciate traditional culture in a more vivid way.
• Artistic Creation AssistanceIt can help artists and designers quickly generate visual elements that match ancient poems during the creative process, providing inspiration and materials.
• Education and LearningIn the field of education, Poetry2Image can be used as a teaching tool to help students better understand and memorize ancient poems and enhance their learning experience by combining images with poems.
• Image generation technology researchPoetry2Image provides a research platform for exploring and improving text-to-image generation techniques, especially when dealing with ancient Chinese poems with rich cultural connotations and complex semantics.