Test-Time Model Adaptation for Image Reconstruction Using Self-supervised Adaptive Layers | SpringerLink
Skip to main content
Springer Nature Link
Log in

Test-Time Model Adaptation for Image Reconstruction Using Self-supervised Adaptive Layers

  • Conference paper
  • First Online:
Computer Vision – ECCV 2024 (ECCV 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15095))

Included in the following conference series:

  • 321 Accesses

Abstract

Image reconstruction from incomplete measurements is a basic task in medical imaging. While supervised deep learning proves to be a powerful tool for image reconstruction, it demands a substantial number of latent images for training. To extend the application of deep learning to medical imaging where collecting latent images poses challenges, this paper introduces an self-supervised test-time adaptation approach. The proposed approach leverages a pre-trained model on an external dataset and efficiently adapts it to each test sample for optimal generalization performance. Model adaption for an unrolling network is done with additional lightweight adaptive linear layers, enabling efficient alignment of testing samples with the distribution targeted in the pre-trained model. This approach is inspired by the connection between linear convolutional layer and Wiener filtering. Extensive experiments showed significant performance gain of the proposed method over other unsupervised methods and model adaptation techniques in two medical imaging tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
JPY 3498
Price includes VAT (Japan)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
JPY 9380
Price includes VAT (Japan)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
JPY 11725
Price includes VAT (Japan)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Chen, D., Tachella, J., Davies, M.E.: Equivariant imaging: learning beyond the range space. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4379–4388 (2021)

    Google Scholar 

  2. Chen, D., Tachella, J., Davies, M.E.: Robust equivariant imaging: a fully unsupervised framework for learning to image from noisy and partial measurements. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5647–5656 (2022)

    Google Scholar 

  3. Chi, Z., Wang, Y., Yu, Y., Tang, J.: Test-time fast adaptation for dynamic scene deblurring via meta-auxiliary learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9137–9146 (2021)

    Google Scholar 

  4. Chung, H., Kim, J., Mccann, M.T., Klasky, M.L., Ye, J.C.: Diffusion posterior sampling for general noisy inverse problems. arXiv preprint arXiv:2209.14687 (2022)

  5. Chung, H., Ye, J.C.: Score-based diffusion models for accelerated MRI. Med. Image Anal. 80, 102479 (2022)

    Article  Google Scholar 

  6. Darestani, M.Z., Liu, J., Heckel, R.: Test-time training can close the natural distribution shift performance gap in deep learning based compressed sensing. In: Proceedings of the 39th International Conference on Machine Learning, vol. 162, pp. 4754–4776. PMLR (2022)

    Google Scholar 

  7. Diamond, S., Sitzmann, V., Heide, F., Wetzstein, G.: Unrolled optimization with deep priors. arXiv preprint arXiv:1705.08041 (2017)

  8. Ding, Q., Chen, G., Zhang, X., Huang, Q., Ji, H., Gao, H.: Low-dose CT with deep learning regularization via proximal forward-backward splitting. Phys. Med. Biol. 65(12), 125009 (2020)

    Article  Google Scholar 

  9. Ding, Q., Nan, Y., Gao, H., Ji, H.: Deep learning with adaptive hyper-parameters for low-dose CT image reconstruction. IEEE Trans. Comput. Imaging 648–660 (2021)

    Google Scholar 

  10. Feng, C.M., Yang, Z., Chen, G., Xu, Y., Shao, L.: Dual-octave convolution for accelerated parallel MR image reconstruction. In: Proceedings of AAAI Conference on Artificial Intelligence, vol. 35, pp. 116–124 (2021)

    Google Scholar 

  11. Jin, K.H., McCann, M.T., Froustey, E., Unser, M.: Deep convolutional neural network for inverse problems in imaging. IEEE Trans. Image Process. 26(9), 4509–4522 (2017)

    Article  MathSciNet  Google Scholar 

  12. Laine, S., Karras, T., Lehtinen, J., Aila, T.: High-quality self-supervised deep image denoising. In: Advances in Neural Information Processing Systems, vol. 32. Curran Associates, Inc. (2019)

    Google Scholar 

  13. Lee, H., Lee, J., Kim, H., Cho, B., Cho, S.: Deep-neural-network-based sinogram synthesis for sparse-view CT image reconstruction. IEEE Trans. Radiat. Plasma Med. Sci. 3(2), 109–119 (2018)

    Article  Google Scholar 

  14. Lehtinen, J., et al.: Noise2noise: learning image restoration without clean data. In: Proceedings of the 35th International Conference on Machine Learning, vol. 80, pp. 2965–2974. PMLR (2018)

    Google Scholar 

  15. Li, Z., Zhang, T., Wan, P., Zhang, D.: Segan: structure-enhanced generative adversarial network for compressed sensing MRI reconstruction. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 2128–2136 (2023)

    Google Scholar 

  16. Liu, J., Kuang, T., Zhang, X.: Image reconstruction by splitting deep learning regularization from iterative inversion. In: Medical Image Computing and Computer Assisted Intervention, pp. 224–231 (2018)

    Google Scholar 

  17. Mansour, Y., Heckel, R.: Zero-shot noise2noise: efficient image denoising without any data. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14018–14027 (2023)

    Google Scholar 

  18. Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings Eighth IEEE International Conference on Computer Vision, vol. 2, pp. 416–423 (2001)

    Google Scholar 

  19. Meinhardt, T., Moller, M., Hazirbas, C., Cremers, D.: Learning proximal operators: using denoising networks for regularizing inverse imaging problems. In: Proceedings of the IEEE International Conference on Computer Vision (2017)

    Google Scholar 

  20. Metzler, C.A., Mousavi, A., Heckel, R., Baraniuk, R.G.: Unsupervised learning with stein’s unbiased risk estimator. arXiv preprint arXiv:1805.10531 (2018)

  21. Mohan, S., Vincent, J.L., Manzorro, R., Crozier, P., Fernandez-Granda, C., Simoncelli, E.: Adaptive denoising via gaintuning. In: Advances in Neural Information Processing Systems, vol. 34, pp. 23727–23740. Curran Associates, Inc. (2021)

    Google Scholar 

  22. Pang, T., Quan, Y., Ji, H.: Self-supervised Bayesian deep learning for image recovery with applications to compressive sensing. In: Proceedings of European Conference on Computer Vision, pp. 475–491 (2020)

    Google Scholar 

  23. Pang, T., Zheng, H., Quan, Y., Ji, H.: Recorrupted-to-recorrupted: unsupervised deep learning for image denoising. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2043–2052 (2021)

    Google Scholar 

  24. Park, S., Yoo, J., Cho, D., Kim, J., Kim, T.H.: Fast adaptation to super-resolution networks via meta-learning. In: Proceedings of European Conference on Computer Vision, pp. 754–769 (2020)

    Google Scholar 

  25. Qin, X., Quan, Y., Pang, T., Ji, H.: Ground-truth free meta-learning for deep compressive sampling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9947–9956 (2023)

    Google Scholar 

  26. Quan, Y., Chen, M., Pang, T., Ji, H.: Self2self with dropout: learning self-supervised denoising from single image. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2020)

    Google Scholar 

  27. Quan, Y., Chen, Z., Pang, T., Ji, H.: Unsupervised deep learning for phase retrieval via teacher-student distillation. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, pp. 2128–2136 (2023)

    Google Scholar 

  28. Quan, Y., Chen, Z., Zheng, H., Ji, H.: Learning deep non-blind image deconvolution without ground truths. In: Proceedings of European Conference on Computer Vision, pp. 642–659 (2022)

    Google Scholar 

  29. Quan, Y., Qin, X., Pang, T., Ji, H.: Dual-domain self-supervised learning and model adaption for deep compressive imaging. In: Proceedings of European Conference on Computer Vision, pp. 409–426 (2022)

    Google Scholar 

  30. Quan, Y., Qin, X., Pang, T., Ji, H.: Siamese cooperative learning for unsupervised image reconstruction from incomplete measurements. IEEE Trans. Pattern Anal. Mach. Intell. 46(7), 4866–4879 (2024)

    Article  Google Scholar 

  31. Romano, Y., Elad, M., Milanfar, P.: The little engine that could: regularization by denoising (red). SIAM J. Imag. Sci. 10(4), 1804–1844 (2017)

    Article  MathSciNet  Google Scholar 

  32. Sidky, E.Y., Kao, C.M., Pan, X.: Accurate image reconstruction from few-views and limited-angle data in divergent-beam CT. J. Xray Sci. Technol. 14(2), 119–139 (2006)

    Google Scholar 

  33. Song, J., Vahdat, A., Mardani, M., Kautz, J.: Pseudoinverse-guided diffusion models for inverse problems. In: International Conference on Learning Representations (2023)

    Google Scholar 

  34. Song, Y., Shen, L., Xing, L., Ermon, S.: Solving inverse problems in medical imaging with score-based generative models. In: International Conference on Learning Representations (2022)

    Google Scholar 

  35. Sun, Y., Wang, X., Liu, Z., Miller, J., Efros, A., Hardt, M.: Test-time training with self-supervision for generalization under distribution shifts. In: Proceedings of the 37th International Conference on Machine Learning, vol. 119, pp. 9229–9248. PMLR (2020)

    Google Scholar 

  36. Ulyanov, D., Vedaldi, A., Lempitsky, V.: Deep image prior. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9446–9454 (2018)

    Google Scholar 

  37. Wakin, M.B., et al.: An introduction to compressive sampling. IEEE Signal Process. Mag. 25(2), 21–30 (2008)

    Article  Google Scholar 

  38. Wang, W., Li, J., Ji, H.: Self-supervised deep image restoration via adaptive stochastic gradient langevin dynamics. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1989–1998 (2022)

    Google Scholar 

  39. Wang, Z., Liu, J., Li, G., Han, H.: Blind2unblind: self-supervised image denoising with visible blind spots. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2027–2036 (2022)

    Google Scholar 

  40. Wang, Z., Zhang, H., Cheng, Z., Chen, B., Yuan, X.: Metasci: scalable and adaptive reconstruction for video compressive sensing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2083–2092 (2021)

    Google Scholar 

  41. Wang, Z., Simoncelli, E.P., Bovik, A.C.: Multiscale structural similarity for image quality assessment. In: The Thirty-Seventh Asilomar Conference on Signals, Systems & Computers, vol. 2, pp. 1398–1402. IEEE (2003)

    Google Scholar 

  42. Wu, W., Hu, D., Niu, C., Yu, H., Vardhanabhuti, V., Wang, G.: Drone: dual-domain residual-based optimization network for sparse-view CT reconstruction. IEEE Trans. Med. Imaging 40(11), 3002–3014 (2021)

    Article  Google Scholar 

  43. Wu, Y., Rosca, M., Lillicrap, T.: Deep compressed sensing. In: International Conference on Machine Learning, pp. 6850–6860. PMLR (2019)

    Google Scholar 

  44. Xiang, J., Dong, Y., Yang, Y.: Fista-net: learning a fast iterative shrinkage thresholding network for inverse problems in imaging. IEEE Trans. Med. Imaging 40(5), 1329–1339 (2021)

    Article  Google Scholar 

  45. Yang, Y., Sun, J., Li, H., Xu, Z.: ADMM-CSNet: a deep learning approach for image compressive sensing. IEEE Trans. Pattern Anal. Mach. Intell. 42(3), 521–538 (2018)

    Article  Google Scholar 

  46. Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H.: Restormer: efficient transformer for high-resolution image restoration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5728–5739 (2022)

    Google Scholar 

  47. Zbontar, J., et al.: fastMRI: an open dataset and benchmarks for accelerated MRI. arXiv preprint arXiv:1811.08839 (2018)

  48. Zhang, J., Ghanem, B.: Ista-net: interpretable optimization-inspired deep network for image compressive sensing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1828–1837 (2018)

    Google Scholar 

  49. Zhang, Z., Liang, X., Dong, X., Xie, Y., Cao, G.: A sparse-view CT reconstruction method based on combination of densenet and deconvolution. IEEE Trans. Med. Imaging 37(6), 1407–1417 (2018)

    Article  Google Scholar 

  50. Zhu, M., et al.: Teacher-student network for CT image reconstruction via meta-learning strategy. In: IEEE Nuclear Science Symposium and Medical Imaging Conference (NSS/MIC), pp. 1–3 (2019)

    Google Scholar 

  51. Zhussip, M., Soltanayev, S., Chun, S.Y.: Training deep learning based image denoisers from undersampled measurements without ground truth and without image prior. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10255–10264 (2019)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Mathematics, National University of Singapore, Singapore, 119076, Singapore

    Yutian Zhao, Tianjing Zhang & Hui Ji

Authors
  1. Yutian Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tianjing Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hui Ji
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yutian Zhao .

Editor information

Editors and Affiliations

  1. University of Birmingham, Birmingham, UK

    Aleš Leonardis

  2. University of Trento, Trento, Italy

    Elisa Ricci

  3. Technical University of Darmstadt, Darmstadt, Germany

    Stefan Roth

  4. Princeton University, Princeton, NJ, USA

    Olga Russakovsky

  5. Czech Technical University in Prague, Prague, Czech Republic

    Torsten Sattler

  6. École des Ponts ParisTech, Marne-la-Vallée, France

    Gül Varol

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 16258 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zhao, Y., Zhang, T., Ji, H. (2025). Test-Time Model Adaptation for Image Reconstruction Using Self-supervised Adaptive Layers. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15095. Springer, Cham. https://doi.org/10.1007/978-3-031-72913-3_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-72913-3_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-72912-6

  • Online ISBN: 978-3-031-72913-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics