Abstract
Diffusion models, which revolutionized image generation, are facing challenges related to intellectual property. These challenges arise when a generated image is influenced by copyrighted images from the training data, a plausible scenario in internet-collected data. Hence, pinpointing influential images from the training dataset, a task known as data attribution, becomes crucial for transparency of content origins. We introduce MONTRAGE, a pioneering data attribution method. Unlike existing approaches that analyze the model post-training, MONTRAGE integrates a novel technique to monitor generations throughout the training via internal model representations. It is tailored for customized diffusion models, where training dynamics access is a practical assumption. This approach, coupled with a new loss function, enhances performance while maintaining efficiency. The advantage of MONTRAGE is evaluated in two granularity-levels: Between-concepts and within-concept, outperforming current state-of-the-art methods for high accuracy. This substantiates MONTRAGE’s insights on diffusion models and its contribution towards copyright solutions for AI digital-art.
J. Brokman and O. Hofman—Equal contribution.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
New York Times: AI Image Generators and Copyright Issues.
- 2.
Harvard Business Review: Generative AI and Intellectual Property Challenges.
- 3.
Adobe Blog: FAIR Act to Protect Artists in the Age of AI.
- 4.
Sometimes Markovianity is not admitted, nonetheless, a multi-step gradual process is still the common practice today.
- 5.
The shifted cosine similarity function adjusts the standard cosine similarity range from \([-1,1]\) to \([0,1]\), aligning with the ground truth values for comparison.
- 6.
References
The proof and measurement of association between two things (1961)
Achille, A., Rovere, M., Soatto, S.: Critical learning periods in deep neural networks. arXiv preprint arXiv:1711.08856 (2017)
Brokman, J., Betser, R., Turjeman, R., Berkov, T., Cohen, I., Gilboa, G.: Enhancing neural training via a correlated dynamics model. In: The Twelfth International Conference on Learning Representations (2024). https://openreview.net/forum?id=c9xsaASm9L
Carlini, N., et al.: Extracting training data from diffusion models. In: 32nd USENIX Security Symposium (USENIX Security 2023), pp. 5253–5270 (2023)
Chandramoorthy, N., Loukas, A., Gatmiry, K., Jegelka, S.: On the generalization of learning algorithms that do not converge. Adv. Neural. Inf. Process. Syst. 35, 34241–34257 (2022)
Dhariwal, P., Nichol, A.: Diffusion models beat GANs on image synthesis. Adv. Neural. Inf. Process. Syst. 34, 8780–8794 (2021)
Fan, M., Chen, C., Wang, C., Huang, J.: On the trustworthiness landscape of state-of-the-art generative models: a comprehensive survey. arXiv preprint arXiv:2307.16680 (2023)
Gal, R., et al.: An image is worth one word: personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618 (2022)
Gandikota, R., Orgad, H., Belinkov, Y., Materzyńska, J., Bau, D.: Unified concept editing in diffusion models. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5111–5120 (2024)
Georgiev, K., Vendrow, J., Salman, H., Park, S.M., Madry, A.: The journey, not the destination: how data guides diffusion models. arXiv preprint arXiv:2312.06205 (2023)
Guo, H., Rajani, N., Hase, P., Bansal, M., Xiong, C.: FastIF: scalable influence functions for efficient model interpretation and debugging. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 10333–10350 (2021)
Hara, S., Nitanda, A., Maehara, T.: Data cleansing for models trained with SGD. In: Advances in Neural Information Processing Systems, vol. 32 (2019)
Harden, M.: The artchive. https://www.artchive.com/
Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Adv. Neural. Inf. Process. Syst. 33, 6840–6851 (2020)
Hu, E.J., et al.: LoRA: low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)
Jiang, H.H., et al.: AI art and its impact on artists. In: Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, pp. 363–374 (2023)
Kawar, B., et al.: Imagic: text-based real image editing with diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6007–6017 (2023)
Khanna, R., Kim, B., Ghosh, J., Koyejo, S.: Interpreting black box predictions using fisher kernels. In: The 22nd International Conference on Artificial Intelligence and Statistics, pp. 3382–3390. PMLR (2019)
Koh, P.W., Liang, P.: Understanding black-box predictions via influence functions. In: International Conference on Machine Learning, pp. 1885–1894. PMLR (2017)
Kumari, N., Zhang, B., Wang, S.Y., Shechtman, E., Zhang, R., Zhu, J.Y.: Ablating concepts in text-to-image diffusion models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22691–22702 (2023)
Kumari, N., Zhang, B., Zhang, R., Shechtman, E., Zhu, J.Y.: Multi-concept customization of text-to-image diffusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1931–1941 (2023)
Kwon, Y., Wu, E., Wu, K., Zou, J.: DataInf: efficiently estimating data influence in loRA-tuned LLMs and diffusion models. In: The Twelfth International Conference on Learning Representations (2024). https://openreview.net/forum?id=9m02ib92Wz
Park, S.M., Georgiev, K., Ilyas, A., Leclerc, G., Mądry, A.: TRAK: attributing model behavior at scale. In: Proceedings of the 40th International Conference on Machine Learning, pp. 27074–27113 (2023)
Pruthi, G., Liu, F., Kale, S., Sundararajan, M.: Estimating training data influence by tracing gradient descent. Adv. Neural. Inf. Process. Syst. 33, 19920–19930 (2020)
Radford, A., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning, pp. 8748–8763. PMLR (2021)
Roich, D., Mokady, R., Bermano, A.H., Cohen-Or, D.: Pivotal tuning for latent-based editing of real images. ACM Trans. Graph. (TOG) 42(1), 1–13 (2022)
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684–10695 (2022)
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., Aberman, K.: DreamBooth: fine tuning text-to-image diffusion models for subject-driven generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500–22510 (2023)
Saharia, C., et al.: Photorealistic text-to-image diffusion models with deep language understanding. Adv. Neural. Inf. Process. Syst. 35, 36479–36494 (2022)
Schuhmann, C., et al.: LAION-5B: an open large-scale dataset for training next generation image-text models. Adv. Neural. Inf. Process. Syst. 35, 25278–25294 (2022)
Somepalli, G., Singla, V., Goldblum, M., Geiping, J., Goldstein, T.: Diffusion art or digital forgery? Investigating data replication in diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6048–6058 (2023)
Wang, S.Y., Efros, A.A., Zhu, J.Y., Zhang, R.: Evaluating data attribution for text-to-image models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7192–7203 (2023)
Yang, L., et al.: Diffusion models: a comprehensive survey of methods and applications. ACM Comput. Surv. 56(4), 1–39 (2023)
Zheng, X., Pang, T., Du, C., Jiang, J., Lin, M.: Intriguing properties of data attribution on diffusion models. In: The Twelfth International Conference on Learning Representations (2024). https://openreview.net/forum?id=vKViCoKGcB
Zilly, J., Achille, A., Censi, A., Frazzoli, E.: On plasticity, invariance, and mutually frozen weights in sequential task learning. Adv. Neural. Inf. Process. Syst. 34, 12386–12399 (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Brokman, J. et al. (2025). MONTRAGE: Monitoring Training for Attribution of Generative Diffusion Models. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15133. Springer, Cham. https://doi.org/10.1007/978-3-031-73226-3_1
Download citation
DOI: https://doi.org/10.1007/978-3-031-73226-3_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-73225-6
Online ISBN: 978-3-031-73226-3
eBook Packages: Computer ScienceComputer Science (R0)