FakeShield - Peking University launches a multimodal large language model framework for detecting image forgery.
FakeShield is a multimodal large-scale language model framework developed by researchers at Peking University, capable of detecting and locating image forgeries. The framework can assess the authenticity of images, generate masks of tampered regions, and provide pixel-level and image-based...
What is FakeShield?
FakeShield, developed by researchers at Peking University, is a multimodal large-scale language model framework capable of detecting and locating image forgeries. The framework assesses the authenticity of images, generates masks of tampered regions, and provides judgment criteria based on pixel-level and image-level tampering clues. FakeShield augments existing datasets with GPT-4o to create the Multimodal Tamper Description Dataset (MMTDSet), used to train tamper analysis capabilities. FakeShield comprises two key modules: Domain Label Guided Interpretable Forgery Detection Module (DTE-FDM) and Multimodal Forgery Localization Module (MFLM), responsible for detection and localization tasks, respectively. FakeShield performs exceptionally well in detecting and locating various tampering techniques such as Photoshop, DeepFake, and AIGC editing, providing an interpretable solution superior to traditional methods.
FakeShield's main functions
- Image authenticity assessment: Determine whether an image has been tampered with.
- Tampering with regional location: Generate a mask for the tampered area in the image.
- Analysis of tampering cluesIt provides judgment criteria based on pixel-level and image-level tampering clues.
- Multimodal data processingCombining visual and language models improves the accuracy and interpretability of detection.
The technical principle of FakeShield
- Multimodal framework designFakeShield is based on a multimodal large language model (M-LLM), which integrates visual and textual information to improve the accuracy of detection and localization.
- Dataset AugmentationEnhance the existing IFDL dataset with GPT-4o to create MMTDSet, providing richer training samples.
- Domain tag guidanceIntroducing domain tags helps distinguish different types of tampered data, enhancing the model's ability to identify different types of tampering.
- Interpretability Module: Develop a DTE-FDM module to provide detection basis based on the analysis of image features and the generation of detailed text descriptions.
- fake positioning moduleUsing the MFLM module, combined with visual language features, the tampered area can be accurately located.
FakeShield's project address
- GitHub repository:https://github.com/zhipeixu/FakeShield
- arXiv technical paper:https://arxiv.org/pdf/2410.02761
Application scenarios of FakeShield
- Social media content moderationAutomatically detect and filter manipulated images on social media platforms to prevent the spread of fake news and misleading content.
- Legal Evidence CollectionIn court evidence collection, it is necessary to determine whether image evidence has been tampered with, in order to ensure the authenticity and validity of the evidence.
- News mediaIt helps news organizations verify the authenticity of news photos and videos, and maintain the accuracy and credibility of news reports.
- Copyright protectionIt provides copyright owners with tools to detect and locate images that are used or altered without authorization, thus protecting intellectual property rights.
- Security monitoringIn the field of security monitoring, ensuring the authenticity of surveillance images is crucial to preventing fraud or illegal activities carried out by tampering with images.