LineArt - A design and drawing appearance transfer framework developed by Jilin University and other institutions.

What is LineArt?

LineArt, developed by Jilin University, the Royal Institute of Technology in Sweden, and the Tokyo Institute of Technology, is a high-quality design drawing appearance transfer framework that requires no training. It can transfer complex appearance features to detailed design drawings, assisting in design and artistic creation. LineArt is based on simulating the hierarchical visual cognition process of humans, integrating artistic experience to guide a diffusion model to generate high-fidelity images while accurately preserving the structural details of the design drawings. LineArt consists of two core stages: first, a multi-frequency line fusion module supplements the detailed structural information of the input design drawings; second, a two-part painting process consisting of base layer shaping and surface layer shading, handling lighting and texture features respectively. The LineArt framework does not require precise 3D modeling or network training, facilitating design task execution, and has demonstrated superior performance compared to existing state-of-the-art methods in experiments.

LineArt's main functions

• High-fidelity appearance transferIt accurately transfers complex appearance features (such as materials, textures, lighting, etc.) from reference photos onto design drawings, making the generated image visually highly consistent with the reference photos.
• Maintain structural detailsDuring the transfer of appearance features, the detailed structure of the design drawings should be accurately preserved to avoid loss or deformation of details and ensure that the original intent of the design drawings is fully reflected.
• No need for precise 3D modeling and network trainingIt eliminates the need for precise 3D modeling, physical property specification, or large-scale network training required in traditional methods, reducing the complexity and cost of design tasks and improving design efficiency.
• Support design and artistic creationIt provides designers and artists with powerful tools to quickly generate design previews with specific visual effects, helping them to intuitively evaluate the design effect, select the most suitable material combination, and accelerate the design iteration process.

LineArt's technical principles

• Multi-frequency line fusion moduleInspired by visual representation theory, the input design drawings are supplemented into an ideal sketch containing three levels of information: continuous single lines for region division, double lines to emphasize local details, and discrete soft edge point sets representing implicit spatial gradients and texture information. This multi-layered visual information supplementation provides rich structural details for the subsequent diffusion model, contributing to the generation of more accurate and realistic images.
• The painting process consists of two parts: base layer shaping and surface layer coloring.:
  • Base layer shapingThe process involves extracting brightness control factors from a reference image to simulate the underpainting process in oil painting. These factors provide initial contrast and structural shape, laying the foundation for subsequent lighting effects and texture generation. Based on the mean and variance that influence the initial noise distribution, the overall brightness of the generated image is controlled, establishing a soft link between the generated and reference images to match their lighting properties.
  • Surface coloringThe texture map obtained by decomposing the reference image is globally encoded and then selectively injected into specific attention layers of the U-net architecture to simulate the coloring process of transparent layers in oil painting. This process integrates text features, content features, and appearance features based on a decoupled cross-attention mechanism, allowing the synthesized texture image to guide appearance generation. Soft edges are used as high-frequency guides to ensure that the generated result is aligned with the texture pattern, promoting natural layout changes and achieving accurate material embedding and appearance feature transfer.
• Knowledge-guided structure preservation and appearance transferLineArt integrates human painting knowledge and visual cognitive processes into image generation. It analyzes three levels of the input sketch, treating prominent features as double lines, object block divisions and geometric edges as single-line visual representations, and using discrete low-level visual features (soft edges) to guide spatial gradient representation and subsequent texture generation, accurately reproducing spatial relationships and adapting to the object's high-frequency textures. In terms of appearance transfer, it borrows the "Imprimatura" technique from classical oil painting, dividing the painting process into two stages: underpainting and glazing. These stages respectively process implicit image information (such as light, illumination, and shadow reflection) and features related to texture and color, achieving a high-fidelity appearance transfer effect.

LineArt's project address

• Project official website:https://meaoxixi.github.io/LineArt/
• arXiv technical paper:https://arxiv.org/pdf/2412.11519v1

Applications of LineArt

• Industrial DesignUsed for product appearance design preview and design iteration acceleration, it can be combined with different material textures to quickly generate realistic effects and optimize product design.
• Interior DesignIt enables furniture material matching and spatial effect simulation, helping designers choose suitable materials and enhance the overall style and texture of interior design.
• Fashion DesignIt showcases the effects of clothing fabrics and the combination of styles and materials, providing a reference for fabric selection and pattern design, and enhancing the innovation and competitiveness of clothing design.
• Architectural DesignSimulate the building's exterior materials and detailed design to assess the building's harmony with its environment, optimize architectural design schemes, and improve building quality.
• Animation and Game DesignIt is applied to character and scene design, generating realistic character images and scene effect previews to enhance the visual experience and immersion.