PhotoDoodle - An artistic image editing framework developed by ByteDance in collaboration with the National University of Singapore and other institutions.
PhotoDoodle is an artistic image editing framework jointly developed by the National University of Singapore, Shanghai Jiao Tong University, Beijing University of Posts and Telecommunications, ByteDance, and the Tiamat team. It learns the unique style of artists based on a small number of samples to achieve...
What is PhotoDoodle?
PhotoDoodle is an artistic image editing framework jointly developed by the National University of Singapore, Shanghai Jiao Tong University, Beijing University of Posts and Telecommunications, ByteDance, and the Tiamat team. It learns the unique styles of artists based on a small number of samples to achieve photo doodling. PhotoDoodle employs a two-stage training strategy: pre-training the general image editing model OmniEditor on large-scale data, and then fine-tuning it with a small number of artist-selected before-and-after image pairs to capture specific editing styles. PhotoDoodle introduces a positional encoding reuse mechanism and a noise-free conditional paradigm to ensure seamless integration and consistency between the generated result and the background. PhotoDoodle releases a high-quality dataset containing 6 styles and over 300 samples, providing a benchmark for related research.
PhotoDoodle's main functions
- Artistic Style Learning and ReproductionLearn unique editing styles from a small number of samples provided by artists and apply them to new image editing tasks.
- Decorative element generationIt supports adding decorative elements (such as hand-drawn lines, color blocks, decorative patterns, etc.) to photos, ensuring that these elements blend seamlessly with the background.
- Maintain background consistencyDuring the editing process, the background content of the original photo must be strictly preserved to avoid background distortion or style loss.
- Instruction-driven editingIt controls image editing content based on natural language commands, enabling precise local modifications and stylization.
- High-efficiency style customizationWith the help of low-rank adaptation (LoRA) technology, only 30-50 pairs of samples are needed to quickly adapt to the styles of different artists, reducing training costs.
The technical principles of PhotoDoodle
- OmniEditor pre-trainingThis paper fine-tunes a pre-trained DiT model using a large-scale image editing dataset, transforming it into a general-purpose image editor (OmniEditor). It introduces a positional encoding cloning mechanism and a noise-free conditional paradigm to ensure spatial consistency and background preservation during the editing process. Precise image editing is achieved by combining text commands and image conditions using a multimodal attention (MMA) mechanism.
- EditLoRA fine-tuningBased on the pre-trained OmniEditor, low-rank adaptation (LoRA) fine-tuning is performed using before-and-after image pairs provided by a small number of artists. The model weights are adaptively adjusted based on the low-rank factorization matrix to capture the editing style of specific artists while retaining the general capabilities of the pre-trained model.
- Location-encoded cloning mechanismThe same positional encoding is shared between the source and target images to ensure spatial consistency of the generated results and avoid misalignment between the background and decorative elements.
- Noise-free conditional paradigmThe generation process preserves the noise-free conditions of the source image, preventing background content from being destroyed during iterative denoising and maintaining the details and textures of the original image.
- Conditional flow matching loss function: Optimize the conditional flow matching loss function to guide the model to learn the generation path from noise to the target image, and further improve the editing effect.
PhotoDoodle project address
- GitHub repository:https://github.com/showlab/PhotoDoodle
- HuggingFace model library:https://huggingface.co/nicolaus-huang/PhotoDoodle
- arXiv technical paper:https://arxiv.org/pdf/2502.14397
Application scenarios of PhotoDoodle
- Digital art creationAdd artistic style to photos and quickly generate creative works.
- Commercial DesignQuickly generate design images that match the brand style, improving design efficiency.
- social mediaAdd decorative effects to your personal photos to enhance the appeal of your shared content.
- Arts Education: To assist teaching and help students understand and practice different art styles.
- Entertainment and Interaction: Generate artistic images in real time to enhance the interactive experience.