InstructMove - The University of Tokyo and Adobe jointly launch an instruction-based image editing model.

What is InstructMove?

InstructMove is an instruction-based image editing model jointly developed by the University of Tokyo and Adobe. It learns to manipulate images according to instructions by observing changes in frame pairs within video. The model generates editing instructions describing changes between frame pairs based on multimodal large language models (MLLMs), training to perform complex, non-rigid editing tasks while maintaining content consistency, such as adjusting subject poses, changing facial expressions, and perspectives. InstructMove uses real video frames as its data source, ensuring the naturalness and realism of the content during editing and overcoming the limitations of synthetic datasets for complex editing tasks. InstructMove supports precise local editing based on control mechanisms such as masks, further enhancing its flexibility and practicality in real-world applications.

The main functions of InstructMove

• Non-rigid editingIt can adjust non-rigid features such as the pose and expression of the subject in the image, conforming to the given editing instructions.
• View adjustment: Change the shooting angle of the image according to the instructions, such as moving the camera angle to the left or right, thereby changing the composition and visual effect of the image.
• Rearrange elementsThis tool rearranges or moves elements in an image, such as putting a toy's legs together or making a bird's tail visible, to meet specific editing needs.
• Precise local editingCombined with control mechanisms such as masks, it supports precise local editing of specific areas of an image, achieving more detailed modification effects.

The technical principle of InstructMove

• Dataset Construction
  • Video frame sampling: Sample frame pairs from internet videos to ensure that there are meaningful transformations between frame pairs, such as changes in subject pose, element movement, or camera view adjustment, and obtain a large number of natural and realistic image transformation samples.
  • Multimodal language model generation instructionsUsing multimodal large language models (MLLMs), such as GPT-4o or Pixtral-12B, the differences between sampled frame pairs are analyzed to generate accurate editing instructions.
• Model Architecture and Training
  • Fine-tuning of pre-trained modelsFine-tuning pre-trained text-to-image (T2I) models, such as Stable Diffusion, on constructed datasets.
  • Spatial Condition StrategyA spatial conditional strategy is introduced to stitch the reference image and the noise input along the spatial dimension, instead of the traditional channel stitching.
  • Denoising Network TrainingThe concatenated input is fed into a denoising U-Net network to predict the noise map. The difference between the predicted noise map and the original noise map is calculated, and the model parameters are optimized to accurately denoise and reconstruct the target image according to editing instructions, thus achieving image editing.
• Control mechanism integration
  • Masked guidanceIt supports integration with control mechanisms such as masks to achieve precise local editing. During the inference phase, masks are used to control the editing area, and the updated latent representation is fused with the reference latent representation to modify specific parts of the image.
  • Other space controlIt integrates with controllable diffusion models such as ControlNet, accepts additional visual cues provided by users, such as sketches or skeleton key points, to achieve more complex and precise image editing operations.

InstructMove's project address

• Project official website:ljzycmd.github.io/projects/InstructMove
• arXiv technical paper:https://arxiv.org/pdf/2412.12087v1

Application scenarios of InstructMove

• Film and television post-productionSpecial effects artists adjust the facial expressions of alien creature characters in science fiction movies to better reflect the anger required by the plot.
• Advertising creative designDesigners used this technique to adjust the perspective and background elements of racing cars in car advertisements, highlighting the speed and passion of new models and attracting consumers' attention.
• Interior DesignInterior designers adjust the position of bedside tables and the style of curtains in the bedroom to meet the client's needs for aesthetics and practicality, creating a warm and comfortable sleeping environment.
• Arts EducationThe teacher adjusts the figures' poses in the painting class to help students understand the relationship between action and emotion, and to deepen their understanding of artistic creation.
• Personal photo editingIndividual users can adjust the expressions in party photos to make them look more natural and relaxed, and then share them on social media to get likes and positive feedback from friends.