Time Information Integration Network for Event Cameras
Computer Science

Computer Science ›› 2022, Vol. 49 ›› Issue (5): 43-49.doi: 10.11896/jsjkx.210400047

• Computer Graphics & Multimedia • Previous Articles     Next Articles

Time Information Integration Network for Event Cameras

XU Hua-chi1, SHI Dian-xi1,2,3, CUI Yu-ning2, JING Luo-xi2, LIU Cong2   

  1. 1 National Innovation Institute of Defense Technology,Beijing 100071,China
    2 College of Computer,National University of Defense Technology,Changsha 410073,China
    3 Tianjin Artificial Intelligence Innovation Center,Tianjin 300457,China
  • Received:2021-04-06 Revised:2021-08-04 Online:2022-05-15 Published:2022-05-06
  • About author:XU Hua-chi,born in 1997,postgra-duate.His main research interests include eventcamera,deep learning and object detection.
    SHI Dian-xi,born in 1966,Ph.D,researcher,Ph.D supervisor,is a member of China Computer Federation.His main research interests include artificial intelligence,distributed computing,cloud computing and big data proces-sing etc.
  • Supported by:
    National Key Research and Development Program of China(2017YFB1001901) and Tian-jin Intelligent Manufacturing Special Fund Project(20181108).

PDF (PC)

1138

Abstract: Event cameras are asynchronous sensors that operate in a completely different way from traditional cameras.Rather than catching pictures at a steady rate,event cameras measure light changes (called events) separately for every pixel.As a sequence,it alleviates the problems of traditional cameras in complex light conditions and scenes where objects move at high speed.With the development of convolutional neural networks,learning-based pattern recognition methods have made great progress in visual tasks such as optical flow estimation and target recognition by converting the output of the event camera into a pseudo-ima-ge representation.However,such methods abandon the temporal correlation between the event streams,so that the texture of the pseudo image is not clear enough,and it is difficult to extract the features.The key to solving this problem lies in how to model relevant information between events in the sample.Therefore,a neural network framework based on event stream partition algorithm is proposed,which explicitly integrates the temporal information of event streams.The framework divides the incoming stream of events into several parts,and a weight distribution network assigns different weights to each piece of the streams.Then,the framework uses convolutional neural network to fuse temporal information and extract advanced features.Finally,the input sample is classified.We thoroughly validate the proposed framework on object recognition.Comparison experiments on N-Caltech101 and N-cars datasets show that the proposed framework has a significant improvement in classification accuracy compared with the most advanced existing algorithms.

Key words: Convolutional neural network, Event streams, Fusion, Temporal information, Weight allocation

CLC Number: 

  • TP183
[1]GALLEGO G,DELBRUCK T,ORCHARD G,et al.Event-based vision:A survey[J].arXiv:1904.08405,2019.
[2]SANG Y G,LI R H,LI Y Q,et al.Research on neuromorphic vision sensor and its applications[J].Chinese Journal on Internet of Things,2019,3(4):63-71.
[3]LICHTSTEINER P,POSCH C,DELBRUCK T.A 128×128120 dB 15 μs Latency Asynchronous Temporal Contrast Vision Sensor[J].IEEE Journal of Solid-state Circuits,2008,43(2):566-576.
[4]LI J N,TIAN Y H.Recent Advances in Neuromorphic Vision Sensors:A Survey[J].Chinese Journal of Computers,2021,44(6):1258-1286.
[5]CUI Y,SHI D,ZHANG Y,et al.Selective Feature Network for Object Detection[C]//2020 International Joint Conference on Neural Networks (IJCNN).IEEE,2020:1-8.
[6]CUI Y,SHI D,ZHANG Y,et al.IDNet:A Single-Shot Object Detector Based on Feature Fusion[C]//2020 IEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAI).IEEE,2020:1137-1144.
[7]ORCHARD G,MEYER C,ETIENNE-CUMMINGS R,et al.HFirst:a temporal approach to object recognition[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2015,37(10):2028-2040.
[8]YAN C D,WANG X,ZUO Y F,et al.Visualization and noise reduction algorithm based on event camera[J].Journal of Beijing University of Aeronautics and Astronautics,2021,47(2):342-350.
[9]LAGORCE X,ORCHARD G,GALLUPPI F,et al.Hots:a hierarchy of event-based time-surfaces for pattern recognition[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2016,39(7):1346-1359.
[10]GALLEGO G,LUND J E A,MUEGGLER E,et al.Event-based,6-DOF camera tracking from photometric depth maps[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2017,40(10):2402-2412.
[11]JIANG M,LIU Z,YU L.A Denoising Algorithm for Eent Came-ras Based on Low-Dimensional Manifold Constraint[J].Journal of Signal Processing,2019,35(10):1753-1761.
[12]KIM H,LEUTENEGGER S,DAVISON A J.Real-time 3D reconstruction and 6-DoF tracking with an event camera[C]//European Conference on Computer Vision.Cham:Springer,2016:349-364.
[13]REDMON J,DIVVALA S,GIRSHICK R,et al.You only look once:Unified,real-time object detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:779-788.
[14]SIRONI A,BRAMBILLA M,BOURDIS N,et al.HATS:Histograms of averaged time surfaces for robust event-based object classification[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:1731-1740.
[15]ZHOU X L,LIU Q Q,CHAN S X,et al.Event Camera-based Visual Tracking Algorithms:a Survey[J].Journal of Chinese Computer Systems,2020,41(11):2325-2332.
[16]MAQUEDA A I,LOQUERCIO A,GALLEGO G,et al.Event-based vision meets deep learning on steering prediction for self-driving cars[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:5419-5427.
[17]GEHRIG D,LOQUERCIO A,DERPANIS K G,et al.End-to-end learning of representations for asynchronous event-based data[C]//Proceedings of the IEEE International Conference on Computer Vision.2019:5633-5643.
[18]MA Y Y,YE Z H,LIU K H,et al.Event-based Visual Localization and Mapping Algorithms:A Survey[J].ACTA Automatica Sinica,2020,46:1-11.
[19]ZHU A Z,YUAN L,CHANEY K,et al.EV-FlowNet:Self-supervised optical flow estimation for event-based cameras[J].arXiv:1802.06898,2018.
[20]REBECQ H,HORSTSCHÄFER T,GALLEGO G,et al.Evo:A geometric approach to event-based 6-dof parallel tracking and mapping in real time[J].IEEE Robotics and Automation Letters,2016,2(2):593-600.
[21]REBECQ H,RANFTL R,KOLTUN V,et al.Events-to-video:Bringing modern computer vision to event cameras[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2019:3857-3866.
[22]HE K,ZHANG X,REN S,et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:770-778.
[23]GIRSHICK R.Fast r-cnn[C]//Proceedings of the IEEE International Conference on Computer Vision.2015:1440-1448.
[24]LEE J H,DELBRUCK T,PFEIFFER M.Training deep spiking neural networks using backpropagation[J].Frontiers in Neuroscience,2016,10:508.
[25]PÉREZ-CARRASCO J A,ZHAO B,SERRANO C,et al.Mapping from frame-driven to frame-free event-driven vision systems by low-rate rate coding and coincidence processing－application to feedforward ConvNets[J].IEEE transactions on Pattern Analysis and Machine Intelligence,2013,35(11):2706-2719.
[26]AMIR A,TABA B,BERG D,et al.A low power,fully event-based gesture recognition system[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:7243-7252.
[27]ORCHARD G,BENOSMAN R,ETIENNE-CUMMINGS R,et al.A spiking neural network architecture for visual motion estimation[C]//2013 IEEE Biomedical Circuits and Systems Conference (BioCAS).IEEE,2013:298-301.
[28]NEIL D,PFEIFFER M,LIU S C.Phased lstm:Accelerating recurrent network training for long or event-based sequences[J].Advances in Neural Information Processing Systems,2016,29:3882-3890.
[29]SHRESTHA S B,ORCHARD G.Slayer:Spike layer error reassignment in time[J].arXiv:1810.08646,2018.
[30]ACHARYA J,PADALA V,BASU A.Spiking neural network based region proposal networks for neuromorphic vision sensors[C]//2019 IEEE International Symposium on Circuits and Systems (ISCAS).IEEE,2019:1-5.
[31]ZHU A Z,YUAN L,CHANEY K,et al.Unsupervised event-based learning of optical flow,depth,and egomotion[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2019:989-997.
[32]REBECQ H,RANFTL R,KOLTUN V,et al.High speed and high dynamic range video with an event camera[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2021,43(6):1964-1980.
[33]CANNICI M,CICCONE M,ROMANONI A,et al.A differen-tiable recurrent surface for asynchronous event-based data[C]//European Conference on Computer Vision.Cham:Springer,2020:136-152.
[34]MESSIKOMMER N,GEHRIG D,LOQUERCIO A,et al.Event-based Asynchronous Sparse Convolutional Networks[J].arXiv:2003.09148,2020.
[35]GALLEGO G,FORSTER C,MUEGGLER E,et al.Event-based camera pose tracking using a generative event model[J].arXiv:1510.01972,2015.
[36]TULYAKOV S,FLEURET F,KIEFEL M,et al.Learning anevent sequence embedding for dense event-based deep stereo[C]//Proceedings of the IEEE International Conference on Computer Vision.2019:1527-1537.
[37]HU J,SHEN L,SUN G.Squeeze-and-excitation networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2018:7132-7141.
[38]ORCHARD G,JAYAWANT A,COHEN G K,et al.Converting static image datasets to spiking neuromorphic datasets using saccades[J].Frontiers in Neuroscience,2015,9:437.
[39]LI F F,FERGUS R,PERONA P.One-shot learning of object categories[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2006,28(4):594-611.
[40]POSCH C,MATOLIN D,WOHLGENANNT R.A QVGA 143 dB dynamic range frame-free PWM image sensor with lossless pixel-level video compression and time-domain CDS[J].IEEE Journal of Solid-State Circuits,2010,46(1):259-275.
[41]SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv:1409.1556,2014.
[42]IOFFE S,SZEGEDY C.Batch normalization:Accelerating deep network training by reducing internal covariate shift[J].arXiv:1502.03167,2015.
[43]RUSSAKOVSKY O,DENG J,SU H,et al.Imagenet large scale visual recognition challenge[J].International Journal of Computer Vision,2015,115(3):211-252.
[44]KINGMA D P,BA J.Adam:A method for stochastic optimization[J].arXiv:1412.6980,2014.
[45]RAMESH B,YANG H,ORCHARD G M,et al.Dart:distribution aware retinal transform for event-based cameras[J].IEEE transactions on Pattern Analysis and Machine Intelligence,2020,12(1):2767-2780.
[46]CANNICI M,CICCONE M,ROMANONI A,et al.Asynchro-nous convolutional networks for object detection in neuromorphic cameras[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops.2019.
[47]DENG Y,LI Y,CHEN H.AMAE:Adaptive Motion-AgnosticEncoder for Event-Based Object Classification[J].IEEE Robotics and Automation Letters,2020,5(3):4596-4603.
[1] WU Zi-yi, LI Shao-mei, JIANG Meng-han, ZHANG Jian-peng. Ontology Alignment Method Based on Self-attention [J]. Computer Science, 2022, 49(9): 215-220.
[2] CAO Xiao-wen, LIANG Mei-yu, LU Kang-kang. Fine-grained Semantic Reasoning Based Cross-media Dual-way Adversarial Hashing Learning Model [J]. Computer Science, 2022, 49(9): 123-131.
[3] ZHOU Le-yuan, ZHANG Jian-hua, YUAN Tian-tian, CHEN Sheng-yong. Sequence-to-Sequence Chinese Continuous Sign Language Recognition and Translation with Multi- layer Attention Mechanism Fusion [J]. Computer Science, 2022, 49(9): 155-161.
[4] XU Yong-xin, ZHAO Jun-feng, WANG Ya-sha, XIE Bing, YANG Kai. Temporal Knowledge Graph Representation Learning [J]. Computer Science, 2022, 49(9): 162-171.
[5] QIN Qi-qi, ZHANG Yue-qin, WANG Run-ze, ZHANG Ze-hua. Hierarchical Granulation Recommendation Method Based on Knowledge Graph [J]. Computer Science, 2022, 49(8): 64-69.
[6] CHEN Yong-quan, JIANG Ying. Analysis Method of APP User Behavior Based on Convolutional Neural Network [J]. Computer Science, 2022, 49(8): 78-85.
[7] ZHU Cheng-zhang, HUANG Jia-er, XIAO Ya-long, WANG Han, ZOU Bei-ji. Deep Hash Retrieval Algorithm for Medical Images Based on Attention Mechanism [J]. Computer Science, 2022, 49(8): 113-119.
[8] WEI Kai-xuan, FU Ying. Re-parameterized Multi-scale Fusion Network for Efficient Extreme Low-light Raw Denoising [J]. Computer Science, 2022, 49(8): 120-126.
[9] SHEN Xiang-pei, DING Yan-rui. Multi-detector Fusion-based Depth Correlation Filtering Video Multi-target Tracking Algorithm [J]. Computer Science, 2022, 49(8): 184-190.
[10] YAN Jia-dan, JIA Cai-yan. Text Classification Method Based on Information Fusion of Dual-graph Neural Network [J]. Computer Science, 2022, 49(8): 230-236.
[11] CHEN Ming-xin, ZHANG Jun-bo, LI Tian-rui. Survey on Attacks and Defenses in Federated Learning [J]. Computer Science, 2022, 49(7): 310-323.
[12] ZHANG Yuan, KANG Le, GONG Zhao-hui, ZHANG Zhi-hong. Related Transaction Behavior Detection in Futures Market Based on Bi-LSTM [J]. Computer Science, 2022, 49(7): 31-39.
[13] ZHANG Ying-tao, ZHANG Jie, ZHANG Rui, ZHANG Wen-qiang. Photorealistic Style Transfer Guided by Global Information [J]. Computer Science, 2022, 49(7): 100-105.
[14] DAI Zhao-xia, LI Jin-xin, ZHANG Xiang-dong, XU Xu, MEI Lin, ZHANG Liang. Super-resolution Reconstruction of MRI Based on DNGAN [J]. Computer Science, 2022, 49(7): 113-119.
[15] CHENG Cheng, JIANG Ai-lian. Real-time Semantic Segmentation Method Based on Multi-path Feature Extraction [J]. Computer Science, 2022, 49(7): 120-126.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
Add:18# Honghu West Rd., Yubei ,Chongqing,China    Post Code: 401121
Tel: 023-63500828/023-67039612/023-67039623
E-mail: jsjkx12@163.com
Powered by Beijing Magtech Co., Ltd.