Follow-Your-Click - an open-source image-to-video model from Tencent and others, capable of generating localized animations.

What is Follow-Your-Click?

Follow-Your-Click is an image-to-video (I2V) generation model jointly developed by researchers from Tencent (Hunyuan Team), Tsinghua University, and Hong Kong University of Science and Technology. It allows users to generate localized image animations through simple clicks and brief action prompts, thus converting static images into dynamic videos. This model aims to address the limitation of most existing image-to-video methods, which lack localized animation capabilities and can only move the entire scene.

Follow-Your-Click official website entry

• Official project homepage:https://follow-your-click.github.io/
• arXiv research paper:https://arxiv.org/abs/2403.08268(Note: The source code and demo are expected to be released in April.)
• GitHub repository:https://github.com/mayuelala/FollowYourClick

The main functions of Follow-Your-Click

• Simple and user-friendly interactionFollow-Your-Click provides an intuitive user interface that allows users to specify animation areas with simple clicks and define animation types and actions with brief prompts.
• Local animation generationUsers can animate specific areas of an image by clicking on them. In other words, users can select any part of an image and add dynamic effects, such as making objects smile, sway, or move.
• Multi-object animationThe model supports simultaneous animation of multiple objects in an image, allowing users to create richer and more complex dynamic scenes.
• Brief action promptsUsers only need to provide a brief description of the action, and the model can understand and generate the corresponding animation effect. This simplifies the animation production process, eliminating the need for users to perform complex operations or provide lengthy descriptions.
• High-quality video generationThe model employs advanced technical strategies, such as first-frame masking and optical flow-based motion amplitude control, to ensure that the generated video has high quality and realism.
• Motion speed controlThe model also allows users to control the movement speed of animated objects, enabling precise control to meet different animation needs.

How Follow-Your-Click works

1. User Interaction:Users first select the area of the object they want to animate by clicking on a specific location on the image. This interaction method is simple and intuitive, requiring no complex operations or detailed descriptions from the user.Next, the user is given a short action prompt, such as “shake your body” or “smile”, to specify the action that should be performed on the selected area.
2. Image segmentation:To convert user clicks into region masks that can be used for animation, the framework integrates the SAM (Segment Anything) tool. SAM is a prompting image segmentation tool that can generate high-quality object masks based on user clicks.
3. First frame masking strategy:To improve video generation quality, the framework employs a first-frame masking strategy. During training, a portion of the latent representation of the input image is randomly masked to enhance the model's ability to learn temporal relevance. This method significantly improves the quality of the generated videos.
4. Motion enhancement module:To enable the model to better respond to short action cues, the framework incorporates a motion augmentation module. This module enhances the model's response to action-related vocabulary through a novel cross-attention layer. During training, this module is trained using short action cues, while during inference, these cues are fed into both the motion augmentation module and U-Net's cross-attention module.
5. Motion amplitude control based on optical flow:Traditional motion intensity control relies on adjusting the frames per second (FPS). However, this method cannot precisely control the motion speed of individual objects. To accurately learn motion speed, the framework proposes an optical flow-based motion amplitude control method. By calculating the average amplitude of the optical flow and projecting it into the position embedding, motion intensity can be consistently applied across all frames.
6. Video generation:During the inference phase, the user's click location and brief action cues are used to generate the animated video. The model combines user-specified region masks and action cues to generate a coherent series of animation frames while keeping the rest of the input image static.