Mitigate Replication and Copying in Diffusion Models with Generalized Caption and Dual Fusion Enhancement

Authors: Chenghao Li, Dake Chen, Yuke Zhang, Peter A. Beerel
Published: 2023-09-13 | Updated: 2024-01-23

Source: https://arxiv.org/abs/2309.07254

PDF: https://arxiv.org/pdf/2309.07254

AIにより推定されたラベル

データ生成 評価指標 プライバシー手法

※ こちらのラベルはAIによって自動的に追加されました。そのため、正確でないことがあります。
詳細は文献データベースについてをご覧ください。

Abstract

While diffusion models demonstrate a remarkable capability for generating high-quality images, their tendency to ‘replicate’ training data raises privacy concerns. Although recent research suggests that this replication may stem from the insufficient generalization of training data captions and duplication of training images, effective mitigation strategies remain elusive. To address this gap, our paper first introduces a generality score that measures the caption generality and employ large language model (LLM) to generalize training captions. Subsequently, we leverage generalized captions and propose a novel dual fusion enhancement approach to mitigate the replication of diffusion models. Our empirical results demonstrate that our proposed methods can significantly reduce replication by 43.5 model while maintaining the diversity and quality of generations. Code is available at https://github.com/HowardLi0816/dual-fusion-diffusion.