Abstract
The employment of the event-based synthetic aperture imaging (E-SAI) technique, which has the capability to capture high-frequency light intensity variations, has facilitated its extensive application on scene de-occlusion reconstruction tasks. However, existing methods usually require prior information and have strict restriction of camera motion on SAI acquisition methods. This paper proposes a novel end-to-end refocus-free variable E-SAI de-occlusion image reconstruction approach REDIR, which can align the global and local features of the variable event data and effectively achieve high-resolution imaging of pure event streams. To further improve the reconstruction of the occluded target, we propose a perceptual mask-gated connection module to interlink information between modules, and incorporate a spatial-temporal attention mechanism into the SNN block to enhance target extraction ability of the model. Through extensive experiments, our model achieves state-of-the-art reconstruction quality on the traditional E-SAI dataset without prior information, while verifying the effectiveness of the variable event data feature registration method on our newly introduced V-ESAI dataset, which obviates the reliance on prior knowledge and extends the applicability of SAI acquisition methods by incorporating focus changes, lens rotations, and non-uniform motion.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Balakrishnan, G., Zhao, A., Sabuncu, M.R., Guttag, J., Dalca, A.V.: Voxelmorph: a learning framework for deformable medical image registration. IEEE Trans. Med. Imaging 38(8), 1788–1800 (2019)
Chowdhury, S.S., Lee, C., Roy, K.: Towards understanding the effect of leak in spiking neural networks. Neurocomputing 464, 83–94 (2021)
Delbrück, T., Linares-Barranco, B., Culurciello, E., Posch, C.: Activity-driven, event-based vision sensors. In: Proceedings of 2010 IEEE International Symposium on Circuits and Systems, pp. 2426–2429. IEEE (2010)
Gallego, G., et al.: Event-based vision: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 44(1), 154–180 (2020)
Gallego, G., Rebecq, H., Scaramuzza, D.: A unifying contrast maximization framework for event cameras, with applications to motion, depth, and optical flow estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3867–3876 (2018)
Gu, C., Learned-Miller, E., Sheldon, D., Gallego, G., Bideau, P.: The spatio-temporal poisson point process: a simple model for the alignment of event camera data. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13495–13504 (2021)
Gupta, A., Sharma, P., Ghosh, D., Honkote, V., Ghose, D.: Event-based time-to-contact estimation with depth image fusion. In: Kumar, S., Sharma, H., Balachandran, K., Kim, J.H., Bansal, J.C. (eds.) CIS 2022, pp. 65–77. Springer, Singapore (2023). https://doi.org/10.1007/978-981-19-9379-4_6
Huang, X., Zhang, Y., Xiong, Z.: Progressive spatio-temporal alignment for efficient event-based motion estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1537–1546 (2023)
Jaderberg, M., Simonyan, K., Zisserman, A., et al.: Spatial transformer networks. In: Advances in Neural Information Processing Systems, vol. 28 (2015)
Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_43
Kim, H., Leutenegger, S., Davison, A.J.: Real-time 3D reconstruction and 6-DoF tracking with an event camera. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 349–364. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46466-4_21
Liu, D., Parra, A., Chin, T.J.: Spatiotemporal registration for event-based visual odometry. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021). https://doi.org/10.1109/cvpr46437.2021.00490
Loshchilov, I., Hutter, F.: SGDR: stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 (2016)
Mahendran, A., Vedaldi, A.: Understanding deep image representations by inverting them. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5188–5196 (2015)
Mitrokhin, A., Fermüller, C., Parameshwara, C., Aloimonos, Y.: Event-based moving object detection and tracking. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1–9. IEEE (2018)
Munda, G., Reinbacher, C., Pock, T.: Real-time intensity-image reconstruction for event cameras using manifold regularisation. Int. J. Comput. Vision 126, 1381–1393 (2018)
Neil, D., Pfeiffer, M., Liu, S.C.: Phased LSTM: accelerating recurrent network training for long or event-based sequences. Cornell University - arXiv, Cornell University - arXiv (2016)
Pan, L., Hartley, R., Scheerlinck, C., Liu, M., Yu, X., Dai, Y.: High frame rate video reconstruction based on an event camera. IEEE Trans. Pattern Anal. Mach. Intell. 1 (2020). https://doi.org/10.1109/tpami.2020.3036667
Qi, C., Su, H., Mo, K., Guibas, L.: Pointnet: deep learning on point sets for 3D classification and segmentation. arxiv 2016. arXiv preprint arXiv:1612.00593
Rebecq, H., Gallego, G., Mueggler, E., Scaramuzza, D.: EMVS: event-based multi-view stereo—3D reconstruction with an event camera in real-time. Int. J. Comput. Vis. 1394–1414 (2018). https://doi.org/10.1007/s11263-017-1050-6
Scheerlinck, C., Rebecq, H., Gehrig, D., Barnes, N., Mahony, R.E., Scaramuzza, D.: Fast image reconstruction with an event camera. In: 2020 IEEE Winter Conference on Applications of Computer Vision (WACV) (2020). https://doi.org/10.1109/wacv45572.2020.9093366
Stearns, C.C., Kannappan, K.: Method for 2-D affine transformation of images (1995)
Vaish, V., Wilburn, B., Joshi, N., Levoy, M.: Using plane+ parallax for calibrating dense camera arrays. In: Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004, vol. 1, p. 1. IEEE (2004)
Wang, Y., Wang, L., Yang, J., An, W., Yu, J., Guo, Y.: Spatial-angular interaction for light field image super-resolution. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12368, pp. 290–308. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58592-1_18
Wang, Y., Wu, T., Yang, J., Wang, L., An, W., Guo, Y.: Deoccnet: learning to see through foreground occlusions in light fields. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 118–127 (2020)
Wang, Z., Chen, J., Hoi, S.C.: Deep learning for image super-resolution: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 43(10), 3365–3387 (2020)
Wang, Z., Ng, Y., Goor, P., Mahony, R.: Event camera calibration of per-pixel biased contrast threshold. arXiv, Computer Vision and Pattern Recognition (2020)
Xu, J., Jiang, M., Yu, L., Yang, W., Wang, W.: Robust motion compensation for event cameras with smooth constraint. IEEE Trans. Comput. Imaging 6, 604–614 (2020)
Yao, M., et al.: Attention spiking neural networks. IEEE Trans. Pattern Anal. Mach. Intell. 45(8), 9393–9410 (2023)
Yu, L., Zhang, X., Liao, W., Yang, W., Xia, G.S.: Learning to see through with events. IEEE Trans. Pattern Anal. Mach. Intell. 45(7), 8660–8678 (2023). https://doi.org/10.1109/TPAMI.2022.3227448
Zhang, X., Liao, W., Yu, L., Yang, W., Xia, G.S.: Event-based synthetic aperture imaging with a hybrid network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14235–14244 (2021)
Zhao, S., Dong, Y., Chang, E.I., Xu, Y., et al.: Recursive cascaded networks for unsupervised medical image registration. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10600–10610 (2019)
Zou, Y., Zheng, Y., Takatani, T., Fu, Y.: Learning to reconstruct high speed and high dynamic range videos from events. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2024–2033 (2021)
Acknowledgements
This work was supported in part by Science and Technology Innovation (STI) 2030—Major Projects under Grant 2022ZD0208700, and National Natural Science Foundation of China under Grant 62376264.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Guo, Q., Shi, H., Li, H., Xiao, J., Gao, X. (2025). REDIR: Refocus-Free Event-Based De-occlusion Image Reconstruction. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15138. Springer, Cham. https://doi.org/10.1007/978-3-031-72989-8_24
Download citation
DOI: https://doi.org/10.1007/978-3-031-72989-8_24
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-72988-1
Online ISBN: 978-3-031-72989-8
eBook Packages: Computer ScienceComputer Science (R0)