Follow-Your-Canvas - A high-resolution video extension technology developed in collaboration between Tencent and Tsinghua University.
Follow-Your-Canvas is a high-resolution video extension technology developed by Tencent's Hunyuan team in collaboration with Tsinghua University and other institutions. Through two core designs—distributed processing and layout alignment—it extends video content to any desired size...
What is Follow-Your-Canvas?
Follow-Your-Canvas is a high-resolution video extension technology developed by Tencent's Hunyuan team in collaboration with Tsinghua University and other institutions. Through two core designs—distributed processing and layout alignment—it extends video content to any resolution while maintaining seamless integration with the original video and generating high-quality content. Suitable for high-resolution video extensions, it overcomes GPU memory limitations, handles large-scale video rendering tasks, and achieves optimal results across various resolutions and aspect ratios.
The main functions of Follow-Your-Canvas
- High resolution output: Expand video content to any resolution, such as from 4K to 8K or higher.
- Not limited by video memory sizeIt can handle large-scale video extrapolation tasks without being limited by the size of video memory.
- Spatiotemporal consistencyDuring the expansion process, maintain spatial and temporal consistency of the video to ensure that the final output video quality is consistent with the style of the original video.
- Generate rich new contentGenerates new content in the specified extended area that matches the style of the original video, enhancing the overall visual effect.
- Large-scale video extensionIt performs exceptionally well in large-scale video extension, such as scaling 512×512 resolution video to 1152×2048 (approximately 9 times) resolution.
The technical principle of Follow-Your-Canvas
- Spatial window segmentationThe video is divided into multiple spatial windows, each of which generates content independently, and then they are seamlessly merged. This allows the system to handle videos of any size and resolution without being limited by GPU memory.
- Layout EncoderThe layout encoder extracts the global layout information of the source video and injects it into the generation process of each window to ensure that the newly generated content is consistent with the layout of the original video.
- Relative Region Embedding (RRE)It provides the relative positional relationship between the source video and the target window, further guiding the generation process of each window, generating extended content that is consistent with the original video layout, and improving spatial and temporal consistency.
- Distributed generationThe content generation of each window is processed in parallel on multiple GPUs, and the generated windows are then seamlessly merged into the final video.
The project address for Follow-Your-Canvas
- Project official website:follow-your-canvas.github.io
- GitHub repository:https://github.com/mayuelala/FollowYourCanvas
- arXiv technical paper:https://arxiv.org/pdf/2409.01055
Application scenarios of Follow-Your-Canvas
- Video resizingIt expands portrait videos shot on a mobile phone into landscape videos to suit different playback needs, such as displaying them on social media or watching them on a widescreen TV.
- Ultra-wide screen video generationIt expands regular video into ultra-wide video, providing a wider field of view, suitable for playback in cinemas or on ultra-wide monitors.
- Panoramic video productionBased on video extension technology, it converts ordinary videos into panoramic videos, enhancing the viewing experience. It is suitable for virtual reality (VR) and 360-degree video production.
- High-resolution video outputCapable of handling video output up to 16K resolution, suitable for applications requiring high-definition video, such as film production or high-quality commercials.