SAVE: Protagonist Diversification with Structure Agnostic Video Editing | SpringerLink
Skip to main content
Springer Nature Link
Log in

SAVE: Protagonist Diversification with Structure Agnostic Video Editing

  • Conference paper
  • First Online:
Computer Vision – ECCV 2024 (ECCV 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15138))

Included in the following conference series:

  • 49 Accesses

Abstract

Driven by the upsurge progress in text-to-image (T2I) generation models, text-to-video (T2V) generation has experienced a significant advance as well. Accordingly, tasks such as modifying the object or changing the style in a video have been possible. However, previous works usually work well on trivial and consistent shapes, and easily collapse on a difficult target that has a largely different body shape from the original one. In this paper, we spot the bias problem in the existing video editing method that restricts the range of choices for the new protagonist and attempt to address this issue using the conventional image-level personalization method. We adopt motion personalization that isolates the motion from a single source video and then modifies the protagonist accordingly. To deal with the natural discrepancy between image and video, we propose a motion word with an inflated textual embedding to properly represent the motion in a source video. We also regulate the motion word to attend to proper motion-related areas by introducing a novel pseudo optical flow, efficiently computed from the pre-calculated attention maps. Finally, we decouple the motion from the appearance of the source video with an additional pseudo word. Extensive experiments demonstrate the editing capability of our method, taking a step toward more diverse and extensive video editing. Our project page: https://ldynx.github.io/SAVE/

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 9380
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 11725
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Avrahami, O., Aberman, K., Fried, O., Cohen-Or, D., Lischinski, D.: Break-a-scene: extracting multiple concepts from a single image. arXiv preprint arXiv:2305.16311 (2023)

  2. Bai, J., et al.: Uniedit: a unified tuning-free framework for video motion and appearance editing. arXiv preprint arXiv:2402.13185 (2024)

  3. Bar-Tal, O., Ofri-Amar, D., Fridman, R., Kasten, Y., Dekel, T.: Text2live: text-driven layered image and video editing. In: European Conference on Computer Vision, pp. 707–723. Springer, Heidelberg (2022). https://doi.org/10.1007/978-3-031-19784-0_41

  4. Blattmann, A., et al.: Align your latents: high-resolution video synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22563–22575 (2023)

    Google Scholar 

  5. Chen, H., et al.: Videocrafter2: overcoming data limitations for high-quality video diffusion models (2024)

    Google Scholar 

  6. Esser, P., Chiu, J., Atighehchian, P., Granskog, J., Germanidis, A.: Structure and content-guided video synthesis with diffusion models. arXiv preprint arXiv:2302.03011 (2023)

  7. Gal, R., et al.: An image is worth one word: personalizing text-to-image generation using textual inversion (2023)

    Google Scholar 

  8. Geyer, M., Bar-Tal, O., Bagon, S., Dekel, T.: Tokenflow: consistent diffusion features for consistent video editing. arXiv preprint arXiv:2307.10373 (2023)

  9. Guo, Y., et al.: Animatediff: animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725 (2023)

  10. Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., Cohen-Or, D.: Prompt-to-prompt image editing with cross attention control (2023)

    Google Scholar 

  11. Hessel, J., Holtzman, A., Forbes, M., Bras, R.L., Choi, Y.: Clipscore: a reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718 (2021)

  12. Ho, J., et al.: Imagen video: high definition video generation with diffusion models. arXiv preprint arXiv:2210.02303 (2022)

  13. Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Adv. Neural. Inf. Process. Syst. 33, 6840–6851 (2020)

    Google Scholar 

  14. Hong, W., Ding, M., Zheng, W., Liu, X., Tang, J.: Cogvideo: large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868 (2022)

  15. Huang, Z., Wu, T., Jiang, Y., Chan, K.C., Liu, Z.: Reversion: diffusion-based relation inversion from images. arXiv preprint arXiv:2303.13495 (2023)

  16. Jin, Y., et al.: Video-lavit: unified video-language pre-training with decoupled visual-motional tokenization (2024)

    Google Scholar 

  17. Kara, O., Kurtkaya, B., Yesiltepe, H., Rehg, J.M., Yanardag, P.: Rave: randomized noise shuffling for fast and consistent video editing with diffusion models (2023)

    Google Scholar 

  18. Kirillov, A., et al.: Segment anything. arXiv preprint arXiv:2304.02643 (2023)

  19. Kumari, N., Zhang, B., Zhang, R., Shechtman, E., Zhu, J.Y.: Multi-concept customization of text-to-image diffusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1931–1941 (2023)

    Google Scholar 

  20. Li, X., Ma, C., Yang, X., Yang, M.H.: Vidtome: video token merging for zero-shot video editing (2023)

    Google Scholar 

  21. Liang, F., et al.: Flowvid: taming imperfect optical flows for consistent video-to-video synthesis (2023)

    Google Scholar 

  22. Liu, S., Zhang, Y., Li, W., Lin, Z., Jia, J.: Video-p2p: video editing with cross-attention control. arXiv preprint arXiv:2303.04761 (2023)

  23. Ma, X., et al.: Latte: latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03048 (2024)

  24. Ma, Z., et al.: Magic-me: identity-specific video customized diffusion (2024)

    Google Scholar 

  25. Mokady, R., Hertz, A., Aberman, K., Pritch, Y., Cohen-Or, D.: Null-text inversion for editing real images using guided diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6038–6047 (2023)

    Google Scholar 

  26. Molad, E., et al.: Dreamix: video diffusion models are general video editors. arXiv preprint arXiv:2302.01329 (2023)

  27. Nichol, A., et al.: Glide: towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 (2021)

  28. Perazzi, F., Pont-Tuset, J., McWilliams, B., Van Gool, L., Gross, M., Sorkine-Hornung, A.: A benchmark dataset and evaluation methodology for video object segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 724–732 (2016)

    Google Scholar 

  29. Qi, C., et al.: Fatezero: fusing attentions for zero-shot text-based video editing (2023)

    Google Scholar 

  30. Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.061251(2), 3 (2022)

  31. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684–10695 (2022)

    Google Scholar 

  32. Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., Aberman, K.: Dreambooth: fine tuning text-to-image diffusion models for subject-driven generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500–22510 (2023)

    Google Scholar 

  33. Saharia, C., et al.: Photorealistic text-to-image diffusion models with deep language understanding. Adv. Neural. Inf. Process. Syst. 35, 36479–36494 (2022)

    Google Scholar 

  34. Singer, U., et al.: Make-a-video: text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 (2022)

  35. Sohn, K., et al.: Styledrop: text-to-image generation in any style. arXiv preprint arXiv:2306.00983 (2023)

  36. Song, Y., Sohl-Dickstein, J., Kingma, D.P., Kumar, A., Ermon, S., Poole, B.: Score-based generative modeling through stochastic differential equations (2021)

    Google Scholar 

  37. Teed, Z., Deng, J.: RAFT: recurrent all-pairs field transforms for optical flow. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12347, pp. 402–419. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58536-5_24

    Chapter  Google Scholar 

  38. Tewel, Y., Gal, R., Chechik, G., Atzmon, Y.: Key-locked rank one editing for text-to-image personalization. In: ACM SIGGRAPH 2023 Conference Proceedings, pp. 1–11 (2023)

    Google Scholar 

  39. Wang, W., et al.: Zero-shot video editing using off-the-shelf image diffusion models. arXiv preprint arXiv:2303.17599 (2023)

  40. Wei, Y., Zhang, Y., Ji, Z., Bai, J., Zhang, L., Zuo, W.: Elite: encoding visual concepts into textual embeddings for customized text-to-image generation (2023)

    Google Scholar 

  41. Wu, C., Liang, J., Ji, L., Yang, F., Fang, Y., Jiang, D., Duan, N.: Nüwa: Visual synthesis pre-training for neural visual world creation. In: European Conference on Computer Vision, pp. 720–736. Springer, Heidelberg (2022). https://doi.org/10.1007/978-3-031-19787-1_41

  42. Wu, J.Z., et al.: Tune-a-video: one-shot tuning of image diffusion models for text-to-video generation (2023)

    Google Scholar 

  43. Wu, J.Z., et al.: Cvpr 2023 text guided video editing competition (2023)

    Google Scholar 

  44. Wu, R., Chen, L., Yang, T., Guo, C., Li, C., Zhang, X.: Lamp: learn a motion pattern for few-shot-based video generation. arXiv preprint arXiv:2310.10769 (2023)

  45. Xiao, G., Yin, T., Freeman, W.T., Durand, F., Han, S.: Fastcomposer: tuning-free multi-subject image generation with localized attention. arXiv preprint arXiv:2305.10431 (2023)

  46. Xu, H., Zhang, J., Cai, J., Rezatofighi, H., Tao, D.: Gmflow: learning optical flow via global matching. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8121–8130 (2022)

    Google Scholar 

  47. Yang, S., Zhou, Y., Liu, Z., Loy, C.C.: Rerender a video: zero-shot text-guided video-to-video translation. arXiv preprint arXiv:2306.07954 (2023)

  48. Yuan, H., et al.: Instructvideo: instructing video diffusion models with human feedback (2023)

    Google Scholar 

  49. Zhang, D.J., et al.: Show-1: marrying pixel and latent diffusion models for text-to-video generation. arXiv preprint arXiv:2309.15818 (2023)

  50. Zhang, Y., Wei, Y., Jiang, D., Zhang, X., Zuo, W., Tian, Q.: Controlvideo: training-free controllable text-to-video generation. arXiv preprint arXiv:2305.13077 (2023)

  51. Zhang, Y., Xing, Z., Zeng, Y., Fang, Y., Chen, K.: Pia: your personalized image animator via plug-and-play modules in text-to-image models (2023)

    Google Scholar 

  52. Zhang, Z., Li, B., Nie, X., Han, C., Guo, T., Liu, L.: Towards consistent video editing with text-to-image diffusion models. Adv. Neural Inf. Process. Syst. (2024)

    Google Scholar 

  53. Zhao, R., et al.: Motiondirector: motion customization of text-to-video diffusion models. arXiv preprint arXiv:2310.08465 (2023)

Download references

Acknowledgments

This work was supported by NRF (2021R1A2C3006659), KOCCA (RS-2024-00398320) and IITP (RS-2021-II211343), all funded by the Korean Government.

Author information

Authors and Affiliations

  1. Seoul National University, Seoul, South Korea

    Yeji Song, Wonsik Shin & Nojun Kwak

  2. NAVER Webtoon AI, Seongnam, South Korea

    Junsoo Lee & Jeesoo Kim

Authors
  1. Yeji Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wonsik Shin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Junsoo Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jeesoo Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Nojun Kwak
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Yeji Song or Nojun Kwak .

Editor information

Editors and Affiliations

  1. University of Birmingham, Birmingham, UK

    Aleš Leonardis

  2. University of Trento, Trento, Italy

    Elisa Ricci

  3. Technical University of Darmstadt, Darmstadt, Hessen, Germany

    Stefan Roth

  4. Princeton University, Palo Alto, CA, USA

    Olga Russakovsky

  5. Czech Technical University in Prague, Prague, Czech Republic

    Torsten Sattler

  6. École des Ponts ParisTech, Marne-la-Vallée, France

    Gül Varol

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (zip 123747 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Song, Y., Shin, W., Lee, J., Kim, J., Kwak, N. (2025). SAVE: Protagonist Diversification with Structure Agnostic Video Editing. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15138. Springer, Cham. https://doi.org/10.1007/978-3-031-72989-8_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-72989-8_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-72988-1

  • Online ISBN: 978-3-031-72989-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation