{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T14:51:31Z","timestamp":1740149491655,"version":"3.37.3"},"reference-count":37,"publisher":"MDPI AG","issue":"6","license":[{"start":{"date-parts":[[2022,3,8]],"date-time":"2022-03-08T00:00:00Z","timestamp":1646697600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001691","name":"Japan Society for the Promotion of Science","doi-asserted-by":"publisher","award":["20K23333 and 20J21208"],"id":[{"id":"10.13039\/501100001691","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"In this work, we propose a method for estimating depth for an image of a monocular camera in order to avoid a collision for the autonomous flight of a drone. The highest flight speed of a drone is generally approximate 22.2 m\/s, and long-distant depth information is crucial for autonomous flights since if the long-distance information is not available, the drone flying at high speeds is prone to collisions. However, long-range, measurable depth cameras are too heavy to be equipped on a drone. This work applies Pix2Pix, which is a kind of Conditional Generative Adversarial Nets (CGAN). Pix2Pix generates depth images from a monocular camera. Additionally, this work applies optical flow to enhance the accuracy of depth estimation. In this work, we propose a highly accurate depth estimation method that effectively embeds an optical flow map into a monocular image. The models are trained with taking advantage of AirSim, which is one of the flight simulators. AirSim can take both monocular and depth images over a hundred meter in the virtual environment, and our model generates a depth image that provides the long-distance information than images captured by a common depth camera. We evaluate accuracy and error of our proposed method using test images in AirSim. In addition, the proposed method is utilized for flight simulation to evaluate the effectiveness to collision avoidance. As a result, our proposed method is higher accuracy and lower error than a state of work. Moreover, our proposed method is lower collision than a state of work.<\/jats:p>","DOI":"10.3390\/s22062097","type":"journal-article","created":{"date-parts":[[2022,3,9]],"date-time":"2022-03-09T06:50:53Z","timestamp":1646808653000},"page":"2097","source":"Crossref","is-referenced-by-count":15,"title":["Pix2Pix-Based Monocular Depth Estimation for Drones with Optical Flow on AirSim"],"prefix":"10.3390","volume":"22","author":[{"given":"Tomoyasu","family":"Shimada","sequence":"first","affiliation":[{"name":"Graduate School of Science and Engineering, Ritsumeikan University, Kusatsu 525-8577, Japan"}]},{"given":"Hiroki","family":"Nishikawa","sequence":"additional","affiliation":[{"name":"Graduate School of Science and Engineering, Ritsumeikan University, Kusatsu 525-8577, Japan"},{"name":"Japan Society for the Promotion of Science, Tokyo 102-0083, Japan"}]},{"given":"Xiangbo","family":"Kong","sequence":"additional","affiliation":[{"name":"Graduate School of Science and Engineering, Ritsumeikan University, Kusatsu 525-8577, Japan"}]},{"given":"Hiroyuki","family":"Tomiyama","sequence":"additional","affiliation":[{"name":"Graduate School of Science and Engineering, Ritsumeikan University, Kusatsu 525-8577, Japan"}]}],"member":"1968","published-online":{"date-parts":[[2022,3,8]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Moffatt, A., Platt, E., Mondragon, B., Kwok, A., Uryeu, D., and Bhandari, S. (2020, January 1\u20134). Obstacle Detection and Avoidance System for Small UAVs Using A LiDAR. Proceedings of the IEEE International Conference on Unmanned Aircraft Systems, Athens, Greece.","DOI":"10.1109\/ICUAS48674.2020.9213897"},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Hou, Y., Zhang, Z., Wang, C., Cheng, S., and Ye, D. (2020, January 24\u201326). Research on Vehicle Identification Method and Vehicle Speed Measurement Method Based on Multi-rotor UAV Equipped with LiDAR. Proceedings of the IEEE International Conference on Advanced Electronic Materials, Computers and Software Engineering, Shenzhen, China.","DOI":"10.1109\/AEMCSE50948.2020.00089"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"278","DOI":"10.1109\/70.88137","article-title":"The Vector Field Histogram-Fast Obstacle Avoidance for Mobile Robots","volume":"7","author":"Borenstein","year":"1991","journal-title":"IEEE Trans. Robot. Autom."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Ma, C., Zhou, Y., and Li, Z. (2020, January 20\u201323). A New Simulation Environment Based on AirSim, ROS, and PX4 for Quadcopter Aircrafts. Proceedings of the International Conference on Control, Automation and Robotics, Singapore.","DOI":"10.1109\/ICCAR49639.2020.9108103"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Ma, D., Tran, A., Keti, N., Yanagi, R., Knight, P., Joglekar, K., Tudor, N., Cresta, B., and Bhandari, S. (2019, January 11\u201314). Flight Test Validation of Collision Avoidance System for a Multicopter using Stereoscopic Vision. Proceedings of the International Conference on Unmanned Aircraft Systems, Atlanta, GA, USA.","DOI":"10.1109\/ICUAS.2019.8798023"},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Perez, E., Winger, A., Tran, A., Garcia-Paredes, C., Run, N., Keti, N., Bhandari, S., and Raheja, A. (2018, January 12\u201315). Autonomous Collision Avoidance System for a Multicopter using Stereoscopic Vision. Proceedings of the IEEE International Conference on Unmanned Aircraft Systems, Dallas, TX, USA.","DOI":"10.1109\/ICUAS.2018.8453417"},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Tsuichihara, S., Akita, S., Ike, R., Shigeta, M., Takemura, H., Natori, T., Aikawa, N., Shindo, K., Ide, Y., and Tejima, S. (2019, January 25\u201327). Drone and GPS Sensors-Based Grassland Management Using Deep-Learning Image Segmentation. Proceedings of the International Conference on Robotic Computing, Naples, Italy.","DOI":"10.1109\/IRC.2019.00123"},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Huang, Z.Y., and Lai, Y.C. (2020, January 1\u20134). Image-Based Sense and Avoid of Small Scale UAV Using Deep Learning Approach. Proceedings of the International Conference on Unmanned Aircraft Systems, Athens, Greece.","DOI":"10.1109\/ICUAS48674.2020.9213884"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Bipin, K., Duggal, V., and Madhava Krishna, K. (2015, January 26\u201330). Autonomous Navigation of Generic Monocular Quadcopter in Natural Environment. Proceedings of the IEEE International Conference on Robotics and Automation, Seattle, WA, USA.","DOI":"10.1109\/ICRA.2015.7139308"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Lin, Y.H., Cheng, W.H., Miao, H., Ku, T.H., and Hsieh, Y.H. (2012, January 25\u201330). Single Image Depth Estimation from Image Descriptors. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Kyoto, Japan.","DOI":"10.1109\/ICASSP.2012.6288007"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Atapour-Abarghouei, A., and Breckon, T.P. (2019, January 22\u201325). Monocular Segment-Wise Depth: Monocular Depth Estimation Based on a Semantic Segmentation Prior. Proceedings of the IEEE International Conference on Image Processing, Taipei, Taiwan.","DOI":"10.1109\/ICIP.2019.8803551"},{"key":"ref_12","unstructured":"Shimada, T., Nishikawa, H., Kong, X., and Tomiyama, H. (2021, January 23\u201324). Pix2Pix-Based Depth Estimation from Monocular Images for Dynamic Path Planning of Multirotor on AirSim. Proceedings of the International Symposium on Advanced Technologies and Applications in the Internet of Things, Kusatsu, Japan."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Fraga-Lamas, P., Ramos, L., Mond\u00e9jar-Guerra, V., and Fern\u00e1ndez-Caram\u00e9s, T.M. (2019). A Review on IoT Deep Learning UAV Systems for Autonomous Obstacle Detection and Collision Avoidance. Remote Sens., 11.","DOI":"10.3390\/rs11182144"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Valisetty, R., Haynes, R., Namburu, R., and Lee, M. (2018, January 17\u201320). Machine Learning for US Army UAVs Sustainment: Assessing Effect of Sensor Frequency and Placement on Damage Information in The Ultrasound Signals. Proceedings of the IEEE International Conference on Machine Learning and Applications, Orlando, FL, USA.","DOI":"10.1109\/ICMLA.2018.00032"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Figetakis, E., and Refaey, A. (2021, January 14\u201323). UAV Path Planning Using on-Board Ultrasound Transducer Arrays and Edge Support. Proceedings of the IEEE International Conference on Communications Workshops, Montreal, QC, Canada.","DOI":"10.1109\/ICCWorkshops50388.2021.9473500"},{"key":"ref_16","unstructured":"McGee, T.G., Sengupta, R., and Hedrick, K. (2005, January 18\u201322). Obstacle Detection for Small Autonomous Aircraft using Sky Segmentation. Proceedings of the IEEE International Conference on Robotics and Automation, Barcelona, Spain."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Redding, J., Amin, J., Boskovic, J., Kang, Y., Hedrick, K., Howlett, J., and Poll, S. (2007, January 20\u201323). A Real-Time Obstacle Detection and Reactive Path Planning System for Autonomous Small-Scale Helicopters. Proceedings of the AIAA Guidance, Navigation and Control Conference and Exhibit, Hilton Head, SC, USA.","DOI":"10.2514\/6.2007-6413"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Trinh, L.A., Thang, N.D., Vu, D.H.N., and Hung, T.C. (2015, January 28\u201330). Position Rectification with Depth Camera to Improve Odometry-based Localization. Proceedings of the International Conference on Communications, Management and Telecommunications (ComManTel), DaNang, Vietnam.","DOI":"10.1109\/ComManTel.2015.7394277"},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"14357","DOI":"10.1007\/s11042-018-6694-x","article-title":"Depth Map Prediction from a Single Image with Generative Adversarial Nets","volume":"79","author":"Zhang","year":"2020","journal-title":"Multimed. Tools Appl."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"2024","DOI":"10.1109\/TPAMI.2015.2505283","article-title":"Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields","volume":"38","author":"Liu","year":"2016","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"1490","DOI":"10.1109\/LRA.2018.2800083","article-title":"J-MOD2: Joint Monocular Obstacle Detection and Depth Estimation","volume":"3","author":"Mancini","year":"2018","journal-title":"IEEE Robot. Autom. Lett."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Hatch, K., Mern, J., and Kochenderfer, M. (2020). Obstacle Avoidance Using a Monocular Camera. arXiv.","DOI":"10.2514\/6.2021-0269"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Hou, Q., and Jung, C. (2017, January 11\u201313). Occlusion Robust Light Field Depth Estimation Using Segmentation Guided Bilateral Filtering. Proceedings of the IEEE International Symposium on Multimedia, Taichung, Taiwan.","DOI":"10.1109\/ISM.2017.13"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"1231","DOI":"10.1177\/0278364913491297","article-title":"Vision Meets Robotics: The KITTI Dataset","volume":"32","author":"Geiger","year":"2013","journal-title":"Int. J. Robot. Res."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Silberman, N., Hoiem, D., Kohli, P., and Fergus, R. (2012, January 7\u201313). Indoor Segmentation and Support Inference from RGBD Images. Proceedings of the ECCV 2012, Florence, Italy.","DOI":"10.1007\/978-3-642-33715-4_54"},{"key":"ref_26","unstructured":"Bhat, S.F., Alhashim, I., and Wonka, P. (2021, January 20\u201325). Adabins: Depth Estimation Using Adaptive Bins. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Isola, P., Zhu, J.Y., Zhou, T., and Efros, A.A. (2017, January 21\u201326). Image-to-Image Translation with Conditional Adversarial Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.632"},{"key":"ref_28","unstructured":"Mirza, M., and Osindero, S. (2014). Conditional generative adversarial nets. arXiv."},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Shah, S., Dey, D., Lovett, C., and Kapoor, A. (2017). AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles. Field and Service Robotics, Springer.","DOI":"10.1007\/978-3-319-67361-5_40"},{"key":"ref_30","unstructured":"Lucas, B.D., and Kanade, T. (1981, January 24\u201328). An Iterative Image Registration Technique with an Application to Stereo Vision. Proceedings of the International Joint Conference on Artificial Intelligence, Vancouver, BC, Canada."},{"key":"ref_31","unstructured":"Farneb\u00e4ck, G. (July, January 29). Two-frame Motion Estimation Based on Polynomial Expansion. Proceedings of the Scandinavian Conference on Image Analysis, Halmstad, Sweden."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5\u20139). U-net: Convolutional Networks for Biomedical Image Segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany.","DOI":"10.1007\/978-3-319-24574-4_28"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"103526","DOI":"10.1016\/j.autcon.2020.103526","article-title":"Quantification of Water Inflow in Rock Tunnel Faces via Convolutional Neural Network Approach","volume":"123","author":"Chen","year":"2021","journal-title":"Autom. Constr."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Eigen, D., and Fergus, R. (2015, January 7\u201313). Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-scale convolutional architecture. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.","DOI":"10.1109\/ICCV.2015.304"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Liu, F., Shen, C., and Lin, G. (2015, January 7\u201312). Deep Convolutional Neural Fields for Depth Estimation from a Single Image. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.","DOI":"10.1109\/CVPR.2015.7299152"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Kuznietsov, Y., Stuckler, J., and Leibe, B. (2017, January 21\u201326). Semi-supervised Deep Learning for Monocular Depth Map Prediction. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.","DOI":"10.1109\/CVPR.2017.238"},{"key":"ref_37","unstructured":"Lidar, V. (2022, March 01). Velodyne Lidar Products. Available online: https:\/\/velodynelidar.com\/products\/."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/6\/2097\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,7,27]],"date-time":"2024-07-27T05:44:26Z","timestamp":1722059066000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/6\/2097"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,8]]},"references-count":37,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2022,3]]}},"alternative-id":["s22062097"],"URL":"https:\/\/doi.org\/10.3390\/s22062097","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2022,3,8]]}}}