Structure-preserving image smoothing with semantic cues | The Visual Computer Skip to main content
Springer Nature Link
Log in

Structure-preserving image smoothing with semantic cues

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

The purpose of image smoothing is to smooth out low-contrast textures while preserving meaningful structures. Although this problem has been studied for decades, it still leaves a lot of space to improve. Recently, learning-based edge detectors have superior performance to traditional manually-designed detectors. Based on the edge detection technique, we present a novel optimization-based image smoothing model combining semantic prior and perform \(L_0\) gradient minimization recursively in our framework to refine the result. Our framework combines the advantage of the state-of-the-art edge detector and the ability of \(L_0\) gradient minimization for structure-preserving image smoothing. Moreover, we employ a large number of real-world images and perform various experiments to evaluate our algorithm. Experimental results show that our algorithm outperforms state-of-the-art algorithms, especially in extracting subjectively-meaningful structures.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Notes

  1. high-frequency textures belonging to input color image are wrongly copied to the output refined depth map, leading to visual artifacts. This is a common issue difficult to solve in the depth upsampling community.

References

  1. Baek, J., Jacobs, D.E.: Accelerating spatially varying Gaussian filters. ACM Trans. Graph. (TOG) 29(6), 169 (2010)

    Article  Google Scholar 

  2. Barron, J.T., Malik, J.: Shape, illumination, and reflectance from shading. TPAMI 37(8), 1670–1687 (2015)

    Article  Google Scholar 

  3. Bell, S., Bala, K., Snavely, N.: Intrinsic images in the wild. ACM Trans. Graph. (TOG) 33(4), 159 (2014)

    Article  Google Scholar 

  4. Bi, S., Han, X., Yu, Y.: An l 1 image transform for edge-preserving smoothing and scene-level intrinsic decomposition. ACM Trans. Graph. (TOG) 34(4), 78 (2015)

    Article  Google Scholar 

  5. Cai, B., Xing, X., Xu, X.: Edge/structure preserving smoothing via relativity-of-Gaussian. In: IEEE International Conference on Image Processing (2017)

  6. Bonneel, N., Sunkavalli, K., Tompkin, J., Sun, D., Paris, S., Pfister, H.: Interactive intrinsic video editing. ACM Trans. Graph. (TOG) 33(6), 197 (2014)

    Article  Google Scholar 

  7. Canny, J.: A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 8(6), 679–698 (1986)

    Article  Google Scholar 

  8. Dollár, P., Zitnick, C.L.: Fast edge detection using structured forests. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 37(8), 1558–1570 (2015)

    Article  Google Scholar 

  9. Duda, R.O., Hart, P.E., Stork, D.G.: Pattern Classification and Scene Analysis, vol. 3. Wiley, New York (1973)

    MATH  Google Scholar 

  10. Farbman, Z., Fattal, R., Lischinski, D., Szeliski, R.: Edge-preserving decompositions for multi-scale tone and detail manipulation. ACM Trans. Graph. (TOG) 27(3), 67 (2008)

    Article  Google Scholar 

  11. Fattal, R.: Edge-avoiding wavelets and their applications. ACM Trans. Graph. (TOG) 28(3), 22 (2009)

    Article  Google Scholar 

  12. Fu, X., Zeng, D., Huang, Y., Zhang, X.P., Ding, X.: A weighted variational model for simultaneous reflectance and illumination estimation. In: CVPR (2016)

  13. Gastal, E.S., Oliveira, M.M.: Domain transform for edge-aware image and video processing. ACM Trans. Graph. (TOG) 30(4), 69 (2011)

    Article  Google Scholar 

  14. Guo, X.: Lime: a method for low-light image enhancement. In: ACM International Conference on Multimedia (2016)

  15. He, K., Sun, J., Tang, X.: Guided image filtering. In: ECCV (2010)

  16. He, L., Schaefer, S.: Mesh denoising via l0 minimization. ACM Trans. Graph. (TOG) 32(4), 64 (2013)

    Google Scholar 

  17. Jeon, J., Lee, H., Kang, H., Lee, S.: Scale-aware structure-preserving texture filtering. Comput. Graph. Forum 35, 77–86 (2016)

    Article  Google Scholar 

  18. Jung, C., Yu, S., Kim, J.: Intensity-guided edge-preserving depth upsampling through weighted l0 gradient minimization. J. Vis. Commun. Image Represent. 42, 132–144 (2017)

    Article  Google Scholar 

  19. Kass, M., Solomon, J.: Smoothed local histogram filters. ACM Trans. Graph. (TOG) 29(4), 1–10 (2010)

    Article  Google Scholar 

  20. Kim, Y., Koh, Y.J., Lee, C., Kim, S., Kim, C.S.: Dark image enhancement based onpairwise target contrast and multi-scale detail boosting. In: IEEE International Conference on Image Processing (2015)

  21. Li, L., Guo, X., Feng, W., Zhang, J.: Soft clustering guided image smoothing. In: IEEE International Conference on Multimedia and Expo, pp. 1–6 (2018)

  22. Li, Y., Brown, M.S.: Single image layer separation using relative smoothness. In: CVPR (2014)

  23. Li, Z., Snavely, N.: Learning intrinsic image decomposition from watching the world. In: CVPR (2018)

  24. Liu, W., Chen, X., Yang, J., Wu, Q.: Robust color guided depth map restoration. IEEE Trans. Image Process. (TIP) 26(1), 315–327 (2017)

    Article  MathSciNet  Google Scholar 

  25. Liu, Y., Cheng, M.M., Hu, X., Wang, K., Bai, X.: Richer convolutional features for edge detection. In: CVPR (2017)

  26. Liang, Z., Xu, J., Zhang, D., Cao, Z.,Zhang, L.: A hybrid l1-l0 layer decomposition model for tone mapping. In: CVPR (2018)

  27. Martin, D., Fowlkes, C., Tal, D., Malik, J., et al.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: ICCV vol. 2, pp. 416–423 (2001)

  28. Meka, A., Fox, G., Zollhöfer, M., Richardt, C., Theobalt, C.: Live user-guided intrinsic video for static scene. IEEE Trans. Vis. Comput. Graph. (TVCG) 23(11), 2447–2454 (2017)

    Article  Google Scholar 

  29. Meka, A., Zollhöfer, M., Richardt, C., Theobalt, C.: Live intrinsic video. ACM Trans. Graph. (TOG) 35(4), 109 (2016)

    Article  Google Scholar 

  30. Min, D., Choi, S., Lu, J., Ham, B., Sohn, K., Do, M.N.: Fast global image smoothing based on weighted least squares. IEEE Trans. Image Process. (TIP) 23(12), 5638–5653 (2014)

    Article  MathSciNet  Google Scholar 

  31. Perazzi, F., Krahenbuhl, P., Pritch, Y., Hornung, A.: Saliency filters: contrast based filtering for salient region detection. In: CVPR (2012)

  32. Ran, M., Zelnik-Manor, L., Tal, A.: How to evaluate foreground maps. In: CVPR (2014)

  33. Rother, C., Kiefel, M., Zhang, L., Schölkopf, B., Gehler, P.V.: Recovering intrinsic images with a global sparsity prior on reflectance. In: NIPS (2011)

  34. Shen, J., Yang, X., Jia, Y., Li, X.: Intrinsic images using optimization. In: CVPR (2011)

  35. Shen, J., Yang, X., Li, X., Jia, Y.: Intrinsic image decomposition using optimization and user scribbles. IEEE Trans. Cybern. 43(2), 425–436 (2013)

    Article  Google Scholar 

  36. Silberman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor segmentation and support inference from rgbd images. In: ECCV (2012)

  37. Subr, K., Soler, C., Durand, F.: Edge-preserving multiscale image decomposition based on local extrema. ACM Trans. Graph. (TOG) 28(5), 147 (2009)

    Article  Google Scholar 

  38. Sun, Y., Schaefer, S., Wang, W.: Denoising point sets via l0 minimization. Comput. Aided Geom. Des. 35, 2–15 (2015)

    Article  Google Scholar 

  39. Sun, Y., Schaefer, S., Wang, W.: Image structure retrieval via \(l_0\) minimization. IEEE Trans. Vis. Comput. Graph. (TVCG) 24(7), 2129–2139 (2018)

    Article  Google Scholar 

  40. Tomasi, C., Manduchi, R.: Bilateral filtering for gray and color images. In: ICCV (1998)

  41. Wei, X., Yang, Q., Gong, Y.: Joint contour filtering. Int. J. Comput. Vis. (IJCV) 126(11), 1245–1265 (2018)

    Article  MathSciNet  Google Scholar 

  42. Wu, C., Zollhöfer, M., Nießner, M., Stamminger, M., Izadi, S., Theobalt, C.: Real-time shading-based refinement for consumer depth cameras. ACM Trans. Graph. (TOG) 33(6), 200 (2014)

    Google Scholar 

  43. Xie, S., Tu, Z.: Holistically-nested edge detection. In: ICCV (2015)

  44. Xu, L., Lu, C., Xu, Y., Jia, J.: Image smoothing via \(l_0\) gradient minimization. ACM Trans. Graph. (TOG) 30(6), 174 (2011)

    Google Scholar 

  45. Xu, L., Yan, Q., Xia, Y., Jia, J.: Structure extraction from texture via relative total variation. ACM Trans. Graph. (TOG) 31(6), 139 (2012)

    Google Scholar 

  46. Yan, Q., Xu, L., Shi, J., Jia, J.: Hierarchical saliency detection. In: CVPR (2013)

  47. Yang, Q.: Recursive bilateral filtering. In: ECCV (2012)

  48. Yang, Q.: Semantic filtering. In: CVPR (2016)

  49. Yu, S., Jung, C., Yun, I., Kim, J.: Intensity-guided depth upsampling using edge sparsity and weighted \(l\_ {0}\) gradient minimization. In: IEEE International Conference on Image Processing (2018)

  50. Zhang, F., Dai, L., Xiang, S., Zhang, X.: Segment graph based image filtering: fast structure-preserving smoothing. In: ICCV (2015)

  51. Zhang, J., Sclaroff, S., Zhe, L., Shen, X., Price, B., Mech, R.: Minimum barrier salient object detection at 80 fps. In: ICCV (2015)

  52. Zhang, Q., Shen, X., Xu, L., Jia, J.: Rolling guidance filter. In: ECCV (2014)

  53. Zhang, Q., Yuan, G., Xiao, C., Zhu, L., Zheng, W.S.: High-quality exposure correction of underexposed photos. In: ACM International Conference on Multimedia (2018)

  54. Zhao, Q., Tan, P., Dai, Q., Shen, L., Wu, E., Lin, S.: A closed-form solution to retinex with nonlocal texture constraints. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 34(7), 1437–1444 (2012)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Yunzhangfang Network Technology Co., Ltd., Nanjing, 210000, China

    Linggang Chen

  2. School of Computer Science, Wuhan University, Wuhan, 430072, China

    Gang Fu

Authors
  1. Linggang Chen
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Gang Fu
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Gang Fu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, L., Fu, G. Structure-preserving image smoothing with semantic cues. Vis Comput 36, 2017–2027 (2020). https://doi.org/10.1007/s00371-020-01950-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-020-01950-1

Keywords