Distraction is All You Need: Memory-Efficient Image Immunization against Diffusion-Based Image Editing
Part Of
Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
Start Page
24462
End Page
24471
ISSN
10636919
DOI (of the container)
10.1109/CVPR52733.2024.02309
Date Issued
2024-06-16
Author(s)
DOI
10.1109/CVPR52733.2024.02309
Abstract
Recent text-to-image (T2I) diffusion models have revolutionized image editing by empowering users to control out-comes using natural language. However, the ease of image manipulation has raised ethical concerns, with the poten-tial for malicious use in generating deceptive or harmful content. To address the concerns, we propose an image im-munization approach named semantic attack to protect our images from being manipulated by malicious agents using diffusion models. Our approach focuses on disrupting the semantic understanding of T2I diffusion models regarding specific content. By attacking the cross-attention mecha-nism that encodes image features with text messages during editing, we distract the model's attention regarding the con-tent of our concern. Our semantic attack renders the model uncertain about the areas to edit, resulting in poorly edited images and contradicting the malicious editing attempts. In addition, by shifting the attack target towards intermediate attention maps from the final generated image, our approach substantially diminishes computational burden and alleviates GPU memory constraints in comparison to pre-vious methods. Moreover, we introduce timestep universal gradient updating to create timestep-agnostic perturbations effective across different input noise levels. By treating the full diffusion process as discrete denoising timesteps during the attack, we achieve equivalent or even superior immu-nization efficacy with nearly half the memory consumption of the previous method. Our contributions include a prac-tical and effective approach to safeguard images against malicious editing, and the proposed method offers robust immunization against various image inpainting and editing approaches, showcasing its potential for real-world appli-cations.
Event(s)
2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024
SDGs
Publisher
IEEE
Type
conference paper