ObjectMover - A new image editing model jointly launched by the University of Hong Kong and Adobe.
ObjectMover is a novel image editing model jointly proposed by the University of Hong Kong and Adobe Research. It addresses issues such as inconsistent lighting and shadows, and object distortion that arise when moving, inserting, and removing objects in an image. It...
What is ObjectMover?
ObjectMover is a novel image editing model jointly proposed by the University of Hong Kong and Adobe Research. It addresses issues such as inconsistent lighting and shadows, and object distortion that arise when objects are moved, inserted, or removed from images. Treating object movement as a special case of two-frame video, it leverages the cross-frame consistency learning capability of a pre-trained video generation model and fine-tunes the model to transfer this to image editing tasks. The model employs sequence-to-sequence modeling, taking as input the original image, the target object image, and the instruction map, and outputting a synthesized image after object movement.
Main functions of ObjectMover
- Object movementIt can move objects in an image to a specified location and automatically adjust related physical effects, such as lighting, shadows, and reflections, while maintaining the object's distinctive features.
- Object deletionIt can realistically fill the background of the removed object, without generating irrelevant new objects, and accurately remove the lighting and shadows associated with the object.
- Object insertionIt can accurately preserve the identity characteristics of the inserted object and automatically generate lighting and shadow effects that are consistent with the environment.
The technical principles of ObjectMover
- Video prior transferObjectMover treats object movement tasks as a special case of two-frame video, leveraging the cross-frame consistency learning capabilities of pre-trained video generation models (such as diffusion models). By fine-tuning the model, it transfers this approach from video generation tasks to image editing tasks. It fully utilizes the physical laws and object correspondences learned during video model pre-training to achieve precise lighting synchronization and identity feature preservation in image editing tasks.
- Sequence-to-sequence modelingThe model reconstructs the object movement task into a sequence prediction problem. Inputs include the original image, the target object image, and a command image (labeling the movement position and direction). Output is a synthesized image of the object after movement. This allows the model to better understand and handle changes in lighting and shadows and occlusion relationships at different positions of the object.
- Synthetic dataset constructionDue to a lack of real-world data on large-scale object movement, the research team used modern game engines (such as Unreal Engine) to generate high-quality synthetic data pairs. The data covered scenes with complex lighting, materials, and occlusion, enhancing the diversity and generalization ability of the model training.
- Multi-task learning strategyObjectMover combines four sub-tasks: object movement, removal, insertion, and video data insertion. It is trained on both synthetic and real video data using a unified framework. This improves the model's generalization ability to real-world scenes, making it more adaptable and robust when handling different image editing tasks.
ObjectMover's project address
- Project official website:https://xinyu-andy.github.io/ObjMover/
- arXiv technical paper:https://arxiv.org/pdf/2503.08037
Application scenarios of ObjectMover
- Special effects productionFor complex special effects scenes, such as the disappearance or appearance of objects, ObjectMover can delete and insert objects while maintaining the realism of the scene.
- Virtual scene editingIn virtual reality and game development, it is necessary to flexibly adjust objects in virtual scenes. ObjectMover can be used to move objects in the scene, such as moving a prop from one position to another, while keeping the object's lighting and shadows consistent with the environment.
- Game level designDevelopers can use ObjectMover to quickly adjust the layout of objects in a level, improving the efficiency of level design.
- Product ShowcaseFor product advertising, ObjectMover can be used to place products in different scenarios to showcase different usage scenarios of the products.
- Spatial planningIn architecture and interior design, ObjectMover can be used to move furniture or decorative items to different locations and evaluate different design options.