SPRIGHT - A large visual language dataset focusing on spatial relationships

What is SPRIGHT?

SPRIGHT (SPatially RIGHT) is a large-scale visual-language dataset focusing on spatial relationships, jointly developed by Arizona State University, Intel Labs, Hugging Face, the University of Washington, and other institutions. It addresses the issue of insufficient spatial consistency in existing text-to-image (T2I) models when generating images. The dataset re-describes approximately 6 million images, emphasizing their spatial relationships and significantly increasing the proportion of spatial relationships in the dataset. Through fine-tuning with SPRIGHT, T2I models achieve significant performance improvements in generating spatially accurate images. SPRIGHT, based on a detailed evaluation and analysis process, validates its effectiveness in capturing spatial relationships, providing a rich resource and foundation for future research.

SPRIGHT's main functions

• Enhanced representation of spatial relationshipsThis dataset re-describes images, emphasizing spatial relationships such as "left/right," "top/bottom," and "front/back." It better captures and represents spatial information within images.
• Improve the spatial consistency of the T2I modelThe T2I model, fine-tuned using the SPRIGHT dataset, can generate images that more accurately match the spatial relationships in the text prompts, improving the spatial consistency of the generated images.
• Supports complex image generation tasksThe SPRIGHT dataset contains rich spatial relationship information, which can help models better understand and generate images containing multiple objects and complex spatial layouts.
• Promote the development of visual-language modelsSPRIGHT provides abundant resources and a foundation for researching and developing more advanced visual-language models, driving technological progress in related fields.

SPRIGHT's technical principles

• Dataset Construction:
  • Image sourceThe images in the SPRIGHT dataset are derived from four widely used visual-language datasets, including CC-12M, Segment Anything, COCO, and LAION-Aesthetics.
  • Re-describeImages are re-described using a large language model (such as LLaVA-1.5-13B) to generate synthetic text descriptions with spatial relationships. The descriptions include spatial relationships and emphasize details such as the relative size and position of objects.
• Capturing spatial relationshipsWhen generating descriptions, the model is instructed to describe objects in the image and their relative positions using specific spatial terms (such as "left/right", "above/below", etc.). This allows the generated descriptions to more accurately reflect the spatial structure of the image.
• Dataset ValidationThe quality and accuracy of the descriptions generated by the SPRIGHT dataset are validated based on multi-level evaluation (such as FAITHScore, GPT-4 evaluation, and human annotation). The evaluation ensures the dataset's effectiveness in capturing spatial relationships.
• Model fine-tuningFine-tuning the T2I model using the SPRIGHT dataset, especially training it on images containing a large number of objects, significantly improves the model's spatial consistency. The fine-tuning method allows the model to better understand and generate images that conform to spatial relationships.

SPRIGHT's project address

• Project official website:spright-t2i.github.io
• GitHub repository:https://github.com/SPRIGHT-T2I/SPRIGHT
• HuggingFace model library:https://huggingface.co/SPRIGHT
• arXiv technical paper:https://arxiv.org/pdf/2404.01197

Application scenarios of SPRIGHT

• Image generation and editingDesigners can generate images that meet specific creative needs, such as creating product display images with specific spatial layouts in advertising design, or generating complex scene background images in game development.
• Virtual Reality and Augmented RealityTo create more realistic virtual scenes in virtual reality applications, such as generating buildings and landscapes with accurate spatial relationships in virtual tourism, thereby enhancing the user's sense of immersion.
• Education and TrainingDevelop visual learning tools in the field of education to help students understand spatial concepts through images. For example, in geometry learning, generate graphics with clear spatial relationships to help students master the properties and relationships of geometric shapes.
• Scientific Research and AnalysisGenerate images of cells or tissues with specific spatial relationships in biological research to help researchers analyze the morphology and function of biological structures.