Follow Your Pose - An open-source, fully gesture-controlled video generation framework

What is Follow Your Pose?

Follow Your Pose is an open-source text-to-video generation framework developed by researchers from Tsinghua University, Hong Kong University of Science and Technology, Tencent AI Lab, and the Chinese Academy of Sciences. It allows users to generate videos using text descriptions and specified human poses. The framework employs a two-stage training strategy to generate videos that are highly consistent with the text descriptions and pose sequences, while maintaining the realism and coherence of the human's movements within the video.

Follow Your Pose official website entrance

• Official project homepage:https://follow-your-pose.github.io/
• GitHub repository:https://github.com/mayuelala/FollowYourPose
• Arxiv research paper:https://arxiv.org/abs/2304.01186
• Hugging Face runtime address:https://huggingface.co/spaces/YueMafighting/FollowYourPose
• OpenXLab runtime address:https://openxlab.org.cn/apps/detail/houshaowei/FollowYourPose
• Google Colab runtime address:https://colab.research.google.com/github/mayuelala/FollowYourPose/blob/main/quick_demo.ipynb

Follow Your Pose Features

• Text to video generationUsers can input text descriptions, and the framework will generate corresponding video content based on these descriptions, such as character movements, scene backgrounds, and overall visual style.
• Attitude controlUsers can control the actions of characters in a video by specifying a sequence of poses, allowing for precise control over every detail of the character's movements in the video.
• Temporal coherenceThe framework can generate videos that are coherent in time, ensuring that the actions and scene changes in the video are natural and smooth, without abrupt jumps or flickering.
• Diverse character and background generationThe framework can generate videos with different appearances, styles, and backgrounds, including but not limited to realistic, cartoon, and cyberpunk styles.
• Multi-role video generationThe framework supports the generation of multi-character videos, which can display multiple characters in the same video and specify the identity and actions of each character based on text descriptions.
• Stylized video generationUsers can generate videos with a specific art style by adding style descriptions (such as "cartoon style", "cyberpunk style", etc.).

How Follow Your Pose works

Follow Your Pose works primarily based on a two-stage training process designed to combine text descriptions and pose information to generate videos. Here are the detailed steps of how it works:

1. Phase 1: Pose-Controlled Text-to-Image Generation
   • Attitude encoderFirst, the framework uses a zero-initialized convolutional encoder to learn pose information. This encoder extracts keypoint features from the input pose sequence.
   • Feature InjectionThe extracted pose features are downsampled to different resolutions and injected into the U-Net structure of a pre-trained text-to-image (T2I) model via residual connections. This allows for the introduction of pose control while maintaining the original model's image generation capabilities.
   • trainAt this stage, the model is trained using only pose image pairs, with the aim of learning how to generate images based on text descriptions and pose information.
2. Phase Two: Video Generation
   • Video datasetTo learn temporal coherence, the framework was trained in the second stage using a video dataset without pose annotations (such as HDVLIA).
   • 3D network structureExtend the pre-trained U-Net model into a 3D network to handle video input. This involves expanding the first layer of convolutions into pseudo-3D convolutions and adding a temporal self-attention module to simulate time series.
   • Cross-frame self-attentionTo further improve video coherence, the framework introduces a cross-frame self-attention module, which helps maintain content consistency between video frames.
   • Fine-tuningAt this stage, only parameters related to temporal coherence (such as temporal self-attention and cross-frame self-attention) are updated, while other parameters (such as pseudo-3D convolutional layers and feedforward networks FFN) remain unchanged.
3. Generation process
   • Text and gesture inputDuring the inference phase, the user inputs text describing the appearance and actions of the target character, as well as a sequence of poses representing the action sequence.
   • Video generationThe model generates videos based on these inputs. During the generation process, most of the parameters of the pre-trained stable diffusion model are frozen, and only the modules related to temporal coherence participate in the computation.

Through this two-stage training strategy, Follow Your Pose can effectively learn from readily available datasets to generate videos with high control and temporal coherence.