libcom - Shanghai Jiao Tong University launches open-source image synthesis problem-solving tool

What is libcom?

libcom is an image compositing toolkit developed by the Shanghai Jiao Tong University (BCMI) laboratory. It aims to solve inconsistencies between foreground and background, such as appearance, geometric, and semantic mismatches, to generate realistic composite images. The toolkit includes various image compositing-related tasks, such as image fusion, image harmonization, shadow generation, object placement, generative compositing, and quality assessment. libcom provides a series of efficient algorithm implementations, which are continuously updated. The goal is to achieve this through a simple import command. import libcom By solving problems related to image compositing, users can easily perform complex image compositing operations.

The main functions of libcom

• Image fusionWhen generating composite images, image fusion technology is used to smooth the transition between the foreground and background, reducing visible seams and unnatural boundaries.
• Image Harmonization(Image Harmonization): Adjusts the lighting and color of the foreground to match the lighting and color of the background, enhancing the realism of the composite image.
• Shadow generation(Shadow Generation): Adds appropriate shadows to the foreground object, enhancing the interaction and three-dimensionality between the foreground and background.
• Object placement(Object Placement): Determines the appropriate position and size of the foreground object in the background, taking into account occlusion, depth, and semantic consistency of the scene.
• Generative Synthesis(Generative Composition): Based on a generative model, it directly generates realistic synthetic images from foreground and background images, taking into account the geometric and semantic information of the foreground.

libcom's technical principles

• Traditional image processingIt smooths image boundaries based on techniques such as Poisson editing and Laplacian pyramids.
• Deep learningThe convolutional neural network (CNN) learns image features and performs pixel-level color and lighting adjustments.
• Generative Adversarial Networks (GANs)In generative compositing, GANs are used to generate foregrounds that blend harmoniously with the background and have correct lighting and shadows.
• diffusion model: Generate high-quality synthetic images based on a diffusion model, given noise and limited foreground information.
• Datasets and pre-trained modelsTrain the model on multiple public datasets to ensure that the algorithm works effectively in diverse scenarios.
• Feature extraction and matchingBased on feature extraction techniques, such as BargainNet, evaluate the harmony between the foreground and background in a synthesized image.

libcom's project address

• Project official websitehttps://pypi.org/project/libcom/
• GitHub repository:https://github.com/bcmi/libcom
• arXiv technical paper:https://arxiv.org/pdf/2106.14490
• Experience the demo online:https://bcmi.sjtu.edu.cn/home/niuli/demo_image_composition/

Application scenarios of libcom

• Augmented Reality (AR)In augmented reality applications, image compositing technology seamlessly integrates virtual objects into real-world scenes, providing users with a more immersive experience.
• Artistic CreationArtists and designers use image compositing technology to combine different visual elements to create entirely new artworks or design concept sketches.
• e-commerceOnline retailers showcase how products look in different environments, such as placing furniture models in real-life photos of homes to help customers virtually try them on.
• Film and video productionIn film post-production, image compositing techniques are used to create complex scenes or place actors in backgrounds that would be impossible to actually film.
• Game developmentGame designers use image compositing technology to create game environments or integrate game characters into real-world scenes to increase the game's fun and realism.